[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:27:55 dist-test-slave-j0hf krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:27:55 dist-test-slave-j0hf krb5kdc[399](info): set up 2 sockets
Sep 27 03:27:55 dist-test-slave-j0hf krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 03:27:56.923554 392 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:27:57 dist-test-slave-j0hf krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:27:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:27:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:27:57.044731 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34987
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34987
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:27:57.126477 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:27:57.126713 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:27:57.126744 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:27:57.127939 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:27:57.127979 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:27:57.127993 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:27:57.128006 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:27:57.128018 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:27:57.129451 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:27:57.129665 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:27:57.129858 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:27:57.135607 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:57.135584 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:27:57.135618 412 server_base.cc:1034] running on GCE node
W20260927 03:27:57.135589 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:27:57.136086 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:27:57.136399 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:27:57.137601 412 hybrid_clock.cc:648] HybridClock initialized: now 1790479677137551 us; error 73 us; skew 500 ppm
Sep 27 03:27:57 dist-test-slave-j0hf krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:27:57.140551 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:27:57.141109 412 webserver.cc:460] Webserver started at http://127.0.98.62:39971/ using document root <none> and password file <none>
I20260927 03:27:57.141358 412 fs_manager.cc:359] Metadata directory not provided
I20260927 03:27:57.141426 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:27:57.141578 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:27:57.142724 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "94827b5f3d11451a94ddf970c71291d8"
format_stamp: "Formatted at 2026-09-27 03:27:57 on dist-test-slave-j0hf"
server_key: "ac50c0c07b1e32c688bc16b09cedd4f5"
server_key_iv: "e788c9605a0a5d20d7f5d90a3c4ce45b"
server_key_version: "encryptionkey@0"
I20260927 03:27:57.143119 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "94827b5f3d11451a94ddf970c71291d8"
format_stamp: "Formatted at 2026-09-27 03:27:57 on dist-test-slave-j0hf"
server_key: "ac50c0c07b1e32c688bc16b09cedd4f5"
server_key_iv: "e788c9605a0a5d20d7f5d90a3c4ce45b"
server_key_version: "encryptionkey@0"
I20260927 03:27:57.144661 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:27:57.145538 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:27:57.145759 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:27:57.145867 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "94827b5f3d11451a94ddf970c71291d8"
format_stamp: "Formatted at 2026-09-27 03:27:57 on dist-test-slave-j0hf"
server_key: "ac50c0c07b1e32c688bc16b09cedd4f5"
server_key_iv: "e788c9605a0a5d20d7f5d90a3c4ce45b"
server_key_version: "encryptionkey@0"
I20260927 03:27:57.145946 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:27:57.183400 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:27:57.186053 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:27:57.186246 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:27:57.190647 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34987
I20260927 03:27:57.190703 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34987 every 8 connection(s)
I20260927 03:27:57.191110 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:27:57.192152 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:27:57.194599 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94827b5f3d11451a94ddf970c71291d8: Bootstrap starting.
I20260927 03:27:57.195230 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94827b5f3d11451a94ddf970c71291d8: Neither blocks nor log segments found. Creating new log.
I20260927 03:27:57.195564 480 log.cc:824] T 00000000000000000000000000000000 P 94827b5f3d11451a94ddf970c71291d8: Log is configured to *not* fsync() on all Append() calls
I20260927 03:27:57.196287 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94827b5f3d11451a94ddf970c71291d8: No bootstrap required, opened a new log
I20260927 03:27:57.197865 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94827b5f3d11451a94ddf970c71291d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94827b5f3d11451a94ddf970c71291d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34987 } }
I20260927 03:27:57.198004 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94827b5f3d11451a94ddf970c71291d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:27:57.198026 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94827b5f3d11451a94ddf970c71291d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94827b5f3d11451a94ddf970c71291d8, State: Initialized, Role: FOLLOWER
I20260927 03:27:57.198196 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94827b5f3d11451a94ddf970c71291d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94827b5f3d11451a94ddf970c71291d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34987 } }
I20260927 03:27:57.198257 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94827b5f3d11451a94ddf970c71291d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:27:57.198310 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94827b5f3d11451a94ddf970c71291d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:27:57.198361 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94827b5f3d11451a94ddf970c71291d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:27:57.198931 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94827b5f3d11451a94ddf970c71291d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94827b5f3d11451a94ddf970c71291d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34987 } }
I20260927 03:27:57.199041 480 leader_election.cc:302] T 00000000000000000000000000000000 P 94827b5f3d11451a94ddf970c71291d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94827b5f3d11451a94ddf970c71291d8; no voters:
I20260927 03:27:57.199234 480 leader_election.cc:288] T 00000000000000000000000000000000 P 94827b5f3d11451a94ddf970c71291d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:27:57.199276 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94827b5f3d11451a94ddf970c71291d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:27:57.199458 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94827b5f3d11451a94ddf970c71291d8 [term 1 LEADER]: Becoming Leader. State: Replica: 94827b5f3d11451a94ddf970c71291d8, State: Running, Role: LEADER
I20260927 03:27:57.199543 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94827b5f3d11451a94ddf970c71291d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:27:57.199558 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94827b5f3d11451a94ddf970c71291d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94827b5f3d11451a94ddf970c71291d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34987 } }
I20260927 03:27:57.200115 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94827b5f3d11451a94ddf970c71291d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94827b5f3d11451a94ddf970c71291d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94827b5f3d11451a94ddf970c71291d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34987 } } }
I20260927 03:27:57.200199 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94827b5f3d11451a94ddf970c71291d8 [sys.catalog]: This master's current role is: LEADER
I20260927 03:27:57.200273 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 412
I20260927 03:27:57.200371 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:27:57.200362 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94827b5f3d11451a94ddf970c71291d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94827b5f3d11451a94ddf970c71291d8. Latest consensus state: current_term: 1 leader_uuid: "94827b5f3d11451a94ddf970c71291d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94827b5f3d11451a94ddf970c71291d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34987 } } }
I20260927 03:27:57.200423 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94827b5f3d11451a94ddf970c71291d8 [sys.catalog]: This master's current role is: LEADER
I20260927 03:27:57.201180 392 external_mini_cluster.cc:1428] Setting key 867aeaea513418eca2963c9ab6c7fedf
W20260927 03:27:57.201277 499 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 94827b5f3d11451a94ddf970c71291d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:27:57.201846 499 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:27:57.202263 492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:27:57.202459 492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:27:57.203919 492 catalog_manager.cc:1347] Generated new cluster ID: ec92d75e80c745018e8cf1145a46a78d
I20260927 03:27:57.203982 492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:27:57 dist-test-slave-j0hf krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:27:57.220081 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:57.201943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38140 (local address 127.0.98.62:34987)
0927 03:27:57.202134 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:57.202144 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:57.203197 (+ 1053us) server_negotiation.cc:407] Connection header received
0927 03:27:57.205429 (+ 2232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:57.205437 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:57.205495 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:57.205659 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:57.207631 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:57.209408 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:57.210845 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:57.211487 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:57.211613 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:57.215883 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:57.215888 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:57.215888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:57.215900 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:57.217855 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:57.218366 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:57.218369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:57.218369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:57.218482 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:57.218764 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:57.218765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:57.218765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:57.219064 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:57.219207 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:57.219554 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 03:27:57.219686 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
I20260927 03:27:57.243885 492 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:27:57.244684 492 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:27:57.253553 492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94827b5f3d11451a94ddf970c71291d8: Generated new TSK 0
I20260927 03:27:57.253783 492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:27:57.284541 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34987
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:27:57.375733 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:27:57.376034 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:27:57.376083 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:27:57.377507 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:27:57.377586 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:27:57.377640 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:27:57.379281 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34987
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:27:57.379643 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:27:57.379915 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:27:57.385669 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:57.385695 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:57.385669 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:27:57.386039 506 server_base.cc:1034] running on GCE node
I20260927 03:27:57.386301 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:27:57.386650 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:27:57.387880 506 hybrid_clock.cc:648] HybridClock initialized: now 1790479677387878 us; error 57 us; skew 500 ppm
Sep 27 03:27:57 dist-test-slave-j0hf krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:27:57.391369 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:27:57.392014 506 webserver.cc:460] Webserver started at http://127.0.98.1:41131/ using document root <none> and password file <none>
I20260927 03:27:57.392292 506 fs_manager.cc:359] Metadata directory not provided
I20260927 03:27:57.392370 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:27:57.392551 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:27:57.393775 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-0/data/instance:
uuid: "09dcff6c169e4dad894514db69d13b57"
format_stamp: "Formatted at 2026-09-27 03:27:57 on dist-test-slave-j0hf"
server_key: "aada81da578e96892f39e1be3b114d42"
server_key_iv: "9fa5eba14d7696a941047316938332b5"
server_key_version: "encryptionkey@0"
I20260927 03:27:57.394220 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-0/wal/instance:
uuid: "09dcff6c169e4dad894514db69d13b57"
format_stamp: "Formatted at 2026-09-27 03:27:57 on dist-test-slave-j0hf"
server_key: "aada81da578e96892f39e1be3b114d42"
server_key_iv: "9fa5eba14d7696a941047316938332b5"
server_key_version: "encryptionkey@0"
I20260927 03:27:57.395875 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260927 03:27:57.396807 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:27:57.396996 506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:27:57.397111 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "09dcff6c169e4dad894514db69d13b57"
format_stamp: "Formatted at 2026-09-27 03:27:57 on dist-test-slave-j0hf"
server_key: "aada81da578e96892f39e1be3b114d42"
server_key_iv: "9fa5eba14d7696a941047316938332b5"
server_key_version: "encryptionkey@0"
I20260927 03:27:57.397197 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:27:57.441355 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:27:57.443909 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:27:57.444104 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:27:57.444363 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:27:57.444690 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:27:57.444775 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:27:57.444818 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:27:57.444846 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:27:57.452172 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35241
I20260927 03:27:57.452375 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35241 every 8 connection(s)
I20260927 03:27:57.452654 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:27:57 dist-test-slave-j0hf krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:27:57.461534 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 506
I20260927 03:27:57.461648 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:27:57.461828 392 external_mini_cluster.cc:1428] Setting key 80f0abf07da4bca30513cb94113b6768
I20260927 03:27:57.463538 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:57.453402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53769 (local address 127.0.98.62:34987)
0927 03:27:57.453467 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:57.453469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:57.453809 (+ 340us) server_negotiation.cc:407] Connection header received
0927 03:27:57.454313 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:57.454314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:57.454329 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:57.454389 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:57.455555 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:57.456080 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:57.457136 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:57.457333 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:57.457390 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:57.459836 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:57.459837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:57.459838 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:57.459845 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:57.461567 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:57.462180 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:57.462181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:57.462182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:57.462274 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:57.462594 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:57.462595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:57.462596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:57.462762 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:57.462820 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:57.463285 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 03:27:57.463381 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 03:27:57.463825 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:57.453391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34987 (local address 127.0.98.1:53769)
0927 03:27:57.453650 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:57.453691 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:27:57.454216 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:27:57.454449 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:27:57.454453 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:27:57.455169 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:27:57.455388 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:57.455395 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:57.456171 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:57.456172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:57.457050 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:57.457053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:57.457409 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:57.457409 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:57.457515 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:57.458056 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:27:57.458065 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:27:57.459719 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:27:57.461691 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:57.461702 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:57.461704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:57.462072 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:57.462343 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:57.462344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:57.462344 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:57.462472 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:57.462850 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:27:57.462859 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:27:57.463083 (+ 224us) client_negotiation.cc:769] Sending connection context
0927 03:27:57.463242 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 03:27:57.463473 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":108,"threads_started":1}
I20260927 03:27:57.464656 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34987
I20260927 03:27:57.464799 635 heartbeater.cc:464] Registering TS with master...
I20260927 03:27:57.465030 635 heartbeater.cc:511] Master 127.0.98.62:34987 requested a full tablet report, sending...
I20260927 03:27:57.465685 444 ts_manager.cc:194] Registered new tserver with Master: 09dcff6c169e4dad894514db69d13b57 (127.0.98.1:35241)
I20260927 03:27:57.466274 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53769
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260927 03:27:58.427896 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 0.965s user 0.001s sys 0.003s
I20260927 03:27:58.428275 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34987
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:27:58.467484 635 heartbeater.cc:503] Master 127.0.98.62:34987 was elected leader, sending a full tablet report...
W20260927 03:27:58.511502 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:27:58.511740 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:27:58.511778 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:27:58.512917 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:27:58.512986 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:27:58.513034 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:27:58.514488 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34987
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:27:58.514752 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:27:58.514984 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:27:58.520292 640 server_base.cc:1034] running on GCE node
W20260927 03:27:58.520265 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:58.520263 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:58.520263 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:27:58.520653 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:27:58.520906 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:27:58.522055 640 hybrid_clock.cc:648] HybridClock initialized: now 1790479678522044 us; error 29 us; skew 500 ppm
Sep 27 03:27:58 dist-test-slave-j0hf krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:27:58.524688 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:27:58.525154 640 webserver.cc:460] Webserver started at http://127.0.98.2:34891/ using document root <none> and password file <none>
I20260927 03:27:58.525338 640 fs_manager.cc:359] Metadata directory not provided
I20260927 03:27:58.525377 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:27:58.525465 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:27:58.526481 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-1/data/instance:
uuid: "9ea29f58d7d34c4680f4e7161b9761b9"
format_stamp: "Formatted at 2026-09-27 03:27:58 on dist-test-slave-j0hf"
server_key: "281386a3ba01b3f54338ccda4fd11679"
server_key_iv: "ef77255c4807ab606f4eafe7b18b2947"
server_key_version: "encryptionkey@0"
I20260927 03:27:58.526815 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9ea29f58d7d34c4680f4e7161b9761b9"
format_stamp: "Formatted at 2026-09-27 03:27:58 on dist-test-slave-j0hf"
server_key: "281386a3ba01b3f54338ccda4fd11679"
server_key_iv: "ef77255c4807ab606f4eafe7b18b2947"
server_key_version: "encryptionkey@0"
I20260927 03:27:58.528283 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:27:58.529060 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:27:58.529238 640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:27:58.529348 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "9ea29f58d7d34c4680f4e7161b9761b9"
format_stamp: "Formatted at 2026-09-27 03:27:58 on dist-test-slave-j0hf"
server_key: "281386a3ba01b3f54338ccda4fd11679"
server_key_iv: "ef77255c4807ab606f4eafe7b18b2947"
server_key_version: "encryptionkey@0"
I20260927 03:27:58.529428 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:27:58.553937 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:27:58.556116 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:27:58.556291 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:27:58.556548 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:27:58.556864 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:27:58.556919 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:27:58.556960 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:27:58.557008 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:27:58.563596 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46867
I20260927 03:27:58.563655 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46867 every 8 connection(s)
I20260927 03:27:58.564034 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:27:58 dist-test-slave-j0hf krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:27:58.573427 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:58.564823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35929 (local address 127.0.98.62:34987)
0927 03:27:58.564947 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:58.564955 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:58.565089 (+ 134us) server_negotiation.cc:407] Connection header received
0927 03:27:58.565599 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:58.565600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:58.565610 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:58.565682 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:58.566611 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:58.567149 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:58.568211 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:58.568406 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:58.568459 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:58.570680 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:58.570682 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:58.570682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:58.570689 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:58.572147 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:58.572550 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:58.572550 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:58.572551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:58.572602 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:58.572823 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:58.572824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:58.572824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:58.572932 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:58.572982 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:58.573266 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 03:27:58.573330 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":69,"threads_started":1}
I20260927 03:27:58.573702 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:58.564796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34987 (local address 127.0.98.2:35929)
0927 03:27:58.564959 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:58.565010 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 03:27:58.565509 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:27:58.565739 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:27:58.565742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:27:58.566317 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:27:58.566516 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:58.566521 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:58.567233 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:58.567233 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:58.568135 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:58.568137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:58.568480 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:58.568480 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:58.568584 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:58.569085 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:27:58.569093 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:27:58.570578 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:27:58.572225 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:58.572231 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:58.572234 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:58.572483 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:58.572664 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:58.572664 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:58.572665 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:58.572770 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:58.573013 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:27:58.573019 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:27:58.573144 (+ 125us) client_negotiation.cc:769] Sending connection context
0927 03:27:58.573243 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 03:27:58.573402 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260927 03:27:58.573728 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 640
I20260927 03:27:58.573837 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:27:58.574034 392 external_mini_cluster.cc:1428] Setting key 0239ac89902b99df6912e6f065fb3c53
I20260927 03:27:58.574268 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34987
I20260927 03:27:58.574389 769 heartbeater.cc:464] Registering TS with master...
I20260927 03:27:58.574623 769 heartbeater.cc:511] Master 127.0.98.62:34987 requested a full tablet report, sending...
I20260927 03:27:58.575038 444 ts_manager.cc:194] Registered new tserver with Master: 9ea29f58d7d34c4680f4e7161b9761b9 (127.0.98.2:46867)
I20260927 03:27:58.575456 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35929
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:27:58.599009 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34987
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:27:58.682827 775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:27:58.683058 775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:27:58.683084 775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:27:58.684237 775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:27:58.684279 775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:27:58.684309 775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:27:58.685901 775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34987
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:27:58.686131 775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:27:58.686324 775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:27:58.691675 780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:58.691752 783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:27:58.691761 781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:27:58.691872 775 server_base.cc:1034] running on GCE node
I20260927 03:27:58.692262 775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:27:58.692626 775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:27:58.693815 775 hybrid_clock.cc:648] HybridClock initialized: now 1790479678693811 us; error 45 us; skew 500 ppm
Sep 27 03:27:58 dist-test-slave-j0hf krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:27:58.696682 775 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:27:58.697165 775 webserver.cc:460] Webserver started at http://127.0.98.3:32851/ using document root <none> and password file <none>
I20260927 03:27:58.697397 775 fs_manager.cc:359] Metadata directory not provided
I20260927 03:27:58.697475 775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:27:58.697631 775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:27:58.698709 775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-2/data/instance:
uuid: "1366e07f98ef4362bc37598abd72a3ee"
format_stamp: "Formatted at 2026-09-27 03:27:58 on dist-test-slave-j0hf"
server_key: "a1c95b104ee631cfc682d2970f402620"
server_key_iv: "cf37bda4c42259591a2920f87acd039e"
server_key_version: "encryptionkey@0"
I20260927 03:27:58.699084 775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1366e07f98ef4362bc37598abd72a3ee"
format_stamp: "Formatted at 2026-09-27 03:27:58 on dist-test-slave-j0hf"
server_key: "a1c95b104ee631cfc682d2970f402620"
server_key_iv: "cf37bda4c42259591a2920f87acd039e"
server_key_version: "encryptionkey@0"
I20260927 03:27:58.700773 775 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:27:58.701670 790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:27:58.701835 775 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:27:58.701934 775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "1366e07f98ef4362bc37598abd72a3ee"
format_stamp: "Formatted at 2026-09-27 03:27:58 on dist-test-slave-j0hf"
server_key: "a1c95b104ee631cfc682d2970f402620"
server_key_iv: "cf37bda4c42259591a2920f87acd039e"
server_key_version: "encryptionkey@0"
I20260927 03:27:58.702015 775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:27:58.723102 775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:27:58.725555 775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:27:58.725737 775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:27:58.726029 775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:27:58.726361 775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:27:58.726415 775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:27:58.726452 775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:27:58.726478 775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:27:58.733425 775 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43141
I20260927 03:27:58.733496 903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43141 every 8 connection(s)
I20260927 03:27:58.733861 775 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:27:58.735316 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 775
I20260927 03:27:58.735491 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790479675865103-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:27:58.735683 392 external_mini_cluster.cc:1428] Setting key 8be3713a64cc1be5eca8f8bd256a0c0a
Sep 27 03:27:58 dist-test-slave-j0hf krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:27:58.743830 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:58.734684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35433 (local address 127.0.98.62:34987)
0927 03:27:58.734771 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:58.734774 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:58.734961 (+ 187us) server_negotiation.cc:407] Connection header received
0927 03:27:58.735554 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:58.735556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:58.735568 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:58.735632 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:58.736983 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:58.737532 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:58.738586 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:58.738814 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:58.738901 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:58.741261 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:58.741263 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:58.741263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:58.741270 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:58.742490 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:58.742914 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:58.742915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:58.742915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:58.742966 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:58.743179 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:58.743180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:58.743180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:58.743282 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:58.743331 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:58.743690 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 03:27:58.743761 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 03:27:58.744027 906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:58.734651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34987 (local address 127.0.98.3:35433)
0927 03:27:58.734832 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 03:27:58.734869 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 03:27:58.735430 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:27:58.735709 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:27:58.735714 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:27:58.736593 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:27:58.736887 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:58.736893 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:58.737632 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:58.737633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:58.738491 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:27:58.738493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:27:58.738911 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:27:58.738913 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:27:58.739057 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:58.739676 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:27:58.739684 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:27:58.741171 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:27:58.742600 (+ 1429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:58.742606 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:58.742608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:58.742845 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:58.743026 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:27:58.743027 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:27:58.743027 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:27:58.743125 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:27:58.743379 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:27:58.743385 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:27:58.743572 (+ 187us) client_negotiation.cc:769] Sending connection context
0927 03:27:58.743663 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 03:27:58.743787 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":108,"threads_started":1}
I20260927 03:27:58.744544 904 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34987
I20260927 03:27:58.744642 904 heartbeater.cc:464] Registering TS with master...
I20260927 03:27:58.744783 904 heartbeater.cc:511] Master 127.0.98.62:34987 requested a full tablet report, sending...
I20260927 03:27:58.745144 444 ts_manager.cc:194] Registered new tserver with Master: 1366e07f98ef4362bc37598abd72a3ee (127.0.98.3:43141)
I20260927 03:27:58.745518 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35433
I20260927 03:27:58.747596 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 03:27:58 dist-test-slave-j0hf krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479678, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 03:27:58 dist-test-slave-j0hf krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479678, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 03:27:58.770706 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:58.762739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52828 (local address 127.0.98.1:35241)
0927 03:27:58.762876 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:58.762881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:58.762893 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:27:58.762937 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:58.762939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:58.762955 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:58.763018 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:58.763680 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:58.764383 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:58.765165 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:58.765532 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:58.765597 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:58.767976 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:58.767986 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:58.767986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:58.767994 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:58.769658 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:58.770009 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:58.770013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:58.770013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:58.770073 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:58.770239 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:58.770240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:58.770240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:58.770375 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:58.770516 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:58.770553 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 03:27:58.770617 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
W20260927 03:27:58.771158 548 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52828
Sep 27 03:27:58 dist-test-slave-j0hf krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 03:27:58 dist-test-slave-j0hf krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 03:27:58.794540 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:27:58.787601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52832 (local address 127.0.98.1:35241)
0927 03:27:58.787677 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 03:27:58.787679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:27:58.787689 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:27:58.787778 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:27:58.787778 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:27:58.787792 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:27:58.787861 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:27:58.788445 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:58.788939 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:58.789692 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:27:58.789904 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:27:58.789999 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:27:58.792325 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:27:58.792327 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:27:58.792327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:27:58.792334 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:27:58.793558 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:58.793919 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:58.793919 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:58.793920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:58.793966 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:27:58.794158 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:27:58.794159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:27:58.794159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:27:58.794248 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:27:58.794338 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 03:27:58.794408 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 03:27:58.794467 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 03:27:58.795272 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 506
I20260927 03:27:58.800184 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 640
I20260927 03:27:58.806166 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 775
I20260927 03:27:58.810693 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 412
2026-09-27T03:27:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2952 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:27:58 dist-test-slave-j0hf krb5kdc[918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:27:58 dist-test-slave-j0hf krb5kdc[918](info): set up 2 sockets
Sep 27 03:27:58 dist-test-slave-j0hf krb5kdc[918](info): commencing operation
krb5kdc: starting...
W20260927 03:27:59.851011 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:27:59 dist-test-slave-j0hf krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:27:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:27:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:27:59.974536 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44809
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44809
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:00.057586 931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:00.057888 931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:00.057940 931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:00.059137 931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:28:00.059217 931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:00.059243 931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:00.059271 931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:28:00.059302 931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:28:00.060806 931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:00.061106 931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:00.061340 931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:00.066763 936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:00.066761 937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:00.066745 931 server_base.cc:1034] running on GCE node
W20260927 03:28:00.066740 939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:00.067207 931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:00.067615 931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:00.068781 931 hybrid_clock.cc:648] HybridClock initialized: now 1790479680068759 us; error 39 us; skew 500 ppm
Sep 27 03:28:00 dist-test-slave-j0hf krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:00.071732 931 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:28:00.072263 931 webserver.cc:460] Webserver started at http://127.0.98.62:45125/ using document root <none> and password file <none>
I20260927 03:28:00.072567 931 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:00.072619 931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:00.072780 931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:00.073915 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "692187589fe34b33a617197d4c344e2a"
format_stamp: "Formatted at 2026-09-27 03:28:00 on dist-test-slave-j0hf"
server_key: "73c9943bde2017d9943ba7aa32c6ebcc"
server_key_iv: "cb50926b42311b17043478c4bd63e7ca"
server_key_version: "encryptionkey@0"
I20260927 03:28:00.074296 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "692187589fe34b33a617197d4c344e2a"
format_stamp: "Formatted at 2026-09-27 03:28:00 on dist-test-slave-j0hf"
server_key: "73c9943bde2017d9943ba7aa32c6ebcc"
server_key_iv: "cb50926b42311b17043478c4bd63e7ca"
server_key_version: "encryptionkey@0"
I20260927 03:28:00.075805 931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260927 03:28:00.076622 946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:00.076795 931 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:00.076894 931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "692187589fe34b33a617197d4c344e2a"
format_stamp: "Formatted at 2026-09-27 03:28:00 on dist-test-slave-j0hf"
server_key: "73c9943bde2017d9943ba7aa32c6ebcc"
server_key_iv: "cb50926b42311b17043478c4bd63e7ca"
server_key_version: "encryptionkey@0"
I20260927 03:28:00.076982 931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:00.097069 931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:00.099272 931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:00.099521 931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:00.103727 931 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44809
I20260927 03:28:00.103762 998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44809 every 8 connection(s)
I20260927 03:28:00.104167 931 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:28:00.105126 999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:00.107600 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 692187589fe34b33a617197d4c344e2a: Bootstrap starting.
I20260927 03:28:00.108218 999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 692187589fe34b33a617197d4c344e2a: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:00.108534 999 log.cc:824] T 00000000000000000000000000000000 P 692187589fe34b33a617197d4c344e2a: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:00.109289 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 692187589fe34b33a617197d4c344e2a: No bootstrap required, opened a new log
I20260927 03:28:00.109501 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 931
I20260927 03:28:00.109584 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:28:00.109740 392 external_mini_cluster.cc:1428] Setting key 59e3be11f40a3df3be118d8018ecc1e6
I20260927 03:28:00.111392 999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 692187589fe34b33a617197d4c344e2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692187589fe34b33a617197d4c344e2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44809 } }
I20260927 03:28:00.111644 999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 692187589fe34b33a617197d4c344e2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:00.111732 999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 692187589fe34b33a617197d4c344e2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692187589fe34b33a617197d4c344e2a, State: Initialized, Role: FOLLOWER
I20260927 03:28:00.111887 999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 692187589fe34b33a617197d4c344e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692187589fe34b33a617197d4c344e2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44809 } }
I20260927 03:28:00.111979 999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 692187589fe34b33a617197d4c344e2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:28:00.112015 999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 692187589fe34b33a617197d4c344e2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:28:00.112073 999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 692187589fe34b33a617197d4c344e2a [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:00.113134 999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 692187589fe34b33a617197d4c344e2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692187589fe34b33a617197d4c344e2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44809 } }
I20260927 03:28:00.113224 999 leader_election.cc:302] T 00000000000000000000000000000000 P 692187589fe34b33a617197d4c344e2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 692187589fe34b33a617197d4c344e2a; no voters:
I20260927 03:28:00.113430 999 leader_election.cc:288] T 00000000000000000000000000000000 P 692187589fe34b33a617197d4c344e2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:28:00.113456 1004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 692187589fe34b33a617197d4c344e2a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:00.113571 1004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 692187589fe34b33a617197d4c344e2a [term 1 LEADER]: Becoming Leader. State: Replica: 692187589fe34b33a617197d4c344e2a, State: Running, Role: LEADER
I20260927 03:28:00.113708 999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 692187589fe34b33a617197d4c344e2a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:28:00.113713 1004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 692187589fe34b33a617197d4c344e2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692187589fe34b33a617197d4c344e2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44809 } }
I20260927 03:28:00.114336 1006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 692187589fe34b33a617197d4c344e2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "692187589fe34b33a617197d4c344e2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692187589fe34b33a617197d4c344e2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44809 } } }
I20260927 03:28:00.114439 1006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 692187589fe34b33a617197d4c344e2a [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:00.114341 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 692187589fe34b33a617197d4c344e2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 692187589fe34b33a617197d4c344e2a. Latest consensus state: current_term: 1 leader_uuid: "692187589fe34b33a617197d4c344e2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692187589fe34b33a617197d4c344e2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44809 } } }
I20260927 03:28:00.114743 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 692187589fe34b33a617197d4c344e2a [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:00.114959 1011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:28:00.115525 1011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 03:28:00 dist-test-slave-j0hf krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:00.118283 1011 catalog_manager.cc:1347] Generated new cluster ID: ecf61cde923947f6b5b70687e69cece2
I20260927 03:28:00.118336 1011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:28:00.121959 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:00.110280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42056 (local address 127.0.98.62:44809)
0927 03:28:00.110450 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:00.110458 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:00.110482 (+ 24us) server_negotiation.cc:407] Connection header received
0927 03:28:00.111371 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:00.111377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:00.111468 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:00.111624 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:00.112469 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.113547 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.114539 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.115170 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.115646 (+ 476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:00.118236 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:00.118239 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:00.118239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:00.118249 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:00.120112 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:00.120651 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:00.120654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:00.120655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:00.120808 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:00.121035 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:00.121036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:00.121037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:00.121285 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:00.121457 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:00.121535 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 03:28:00.121811 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260927 03:28:00.140600 1011 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:28:00.141387 1011 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:28:00.146656 1011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 692187589fe34b33a617197d4c344e2a: Generated new TSK 0
I20260927 03:28:00.146864 1011 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:28:00.180402 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44809
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:00.279848 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:00.280186 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:00.280230 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:00.281675 1025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:00.281745 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:00.281793 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:28:00.284082 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44809
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:00.284396 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:00.284641 1025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:00.290022 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:00.290030 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:00.290045 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:00.290279 1025 server_base.cc:1034] running on GCE node
I20260927 03:28:00.290494 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:00.290798 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:00.291991 1025 hybrid_clock.cc:648] HybridClock initialized: now 1790479680291973 us; error 69 us; skew 500 ppm
Sep 27 03:28:00 dist-test-slave-j0hf krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:00.294798 1025 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:28:00.295346 1025 webserver.cc:460] Webserver started at http://127.0.98.1:42249/ using document root <none> and password file <none>
I20260927 03:28:00.295645 1025 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:00.295696 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:00.295835 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:00.297008 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-0/data/instance:
uuid: "3e1c5443fed74fdf8917a4dfdb0dd85b"
format_stamp: "Formatted at 2026-09-27 03:28:00 on dist-test-slave-j0hf"
server_key: "08d9e29d0e7d2f795fc9836c045f46c5"
server_key_iv: "835535dfa86c5bd78c0effd5dfbbf33f"
server_key_version: "encryptionkey@0"
I20260927 03:28:00.297379 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3e1c5443fed74fdf8917a4dfdb0dd85b"
format_stamp: "Formatted at 2026-09-27 03:28:00 on dist-test-slave-j0hf"
server_key: "08d9e29d0e7d2f795fc9836c045f46c5"
server_key_iv: "835535dfa86c5bd78c0effd5dfbbf33f"
server_key_version: "encryptionkey@0"
I20260927 03:28:00.298808 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:28:00.299726 1040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:00.299939 1025 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:00.300040 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "3e1c5443fed74fdf8917a4dfdb0dd85b"
format_stamp: "Formatted at 2026-09-27 03:28:00 on dist-test-slave-j0hf"
server_key: "08d9e29d0e7d2f795fc9836c045f46c5"
server_key_iv: "835535dfa86c5bd78c0effd5dfbbf33f"
server_key_version: "encryptionkey@0"
I20260927 03:28:00.300125 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:00.348524 1025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:00.350781 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:00.350916 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:00.351120 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:00.351549 1025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:00.351608 1025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:00.351655 1025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:00.351686 1025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:00.357949 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37813
I20260927 03:28:00.357993 1153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37813 every 8 connection(s)
I20260927 03:28:00.358325 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:28:00 dist-test-slave-j0hf krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:00.367632 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 1025
I20260927 03:28:00.367722 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:28:00.367751 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:00.359108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54609 (local address 127.0.98.62:44809)
0927 03:28:00.359171 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:00.359174 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:00.359416 (+ 242us) server_negotiation.cc:407] Connection header received
0927 03:28:00.359973 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:00.359974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:00.359983 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:00.360046 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:00.360957 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.361456 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.362466 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.362659 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.362712 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:00.364999 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:00.365001 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:00.365001 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:00.365010 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:00.366436 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:00.366799 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:00.366800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:00.366800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:00.366848 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:00.367065 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:00.367066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:00.367066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:00.367173 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:00.367223 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:00.367545 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 03:28:00.367637 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 03:28:00.367899 392 external_mini_cluster.cc:1428] Setting key 22f3c8b72457055375e3a9462e756cef
I20260927 03:28:00.367941 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:00.359100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44809 (local address 127.0.98.1:54609)
0927 03:28:00.359269 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:00.359313 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:00.359886 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:00.360102 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:00.360106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:00.360678 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:00.360878 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:00.360883 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.361546 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.361547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:00.362360 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:00.362362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.362751 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.362752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:00.362869 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:00.363361 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:00.363369 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:00.364901 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:00.366519 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:00.366524 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:00.366525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:00.366734 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:00.366913 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:00.366914 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:00.366914 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:00.367010 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:00.367228 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:00.367231 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:00.367350 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 03:28:00.367515 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 03:28:00.367650 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
I20260927 03:28:00.368553 1154 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44809
I20260927 03:28:00.368662 1154 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:00.368837 1154 heartbeater.cc:511] Master 127.0.98.62:44809 requested a full tablet report, sending...
I20260927 03:28:00.369366 963 ts_manager.cc:194] Registered new tserver with Master: 3e1c5443fed74fdf8917a4dfdb0dd85b (127.0.98.1:37813)
I20260927 03:28:00.370107 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54609
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:28:00.394455 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44809
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:00.477900 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:00.478147 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:00.478175 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:00.479588 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:00.479641 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:00.479674 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:28:00.481325 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44809
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:00.481542 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:00.481739 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:00.486970 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:00.487000 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:00.487000 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:00.487039 1159 server_base.cc:1034] running on GCE node
I20260927 03:28:00.487397 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:00.487694 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:00.488870 1159 hybrid_clock.cc:648] HybridClock initialized: now 1790479680488858 us; error 29 us; skew 500 ppm
Sep 27 03:28:00 dist-test-slave-j0hf krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:00.491343 1159 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:28:00.491835 1159 webserver.cc:460] Webserver started at http://127.0.98.2:36751/ using document root <none> and password file <none>
I20260927 03:28:00.492043 1159 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:00.492084 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:00.492178 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:00.493193 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-1/data/instance:
uuid: "97886abebfed42ef9df9dc81cd4c45fb"
format_stamp: "Formatted at 2026-09-27 03:28:00 on dist-test-slave-j0hf"
server_key: "9c6dbc187abf503167031180c04fdf71"
server_key_iv: "2218ac92f9c0edd6bfc668249c0c87db"
server_key_version: "encryptionkey@0"
I20260927 03:28:00.493510 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-1/wal/instance:
uuid: "97886abebfed42ef9df9dc81cd4c45fb"
format_stamp: "Formatted at 2026-09-27 03:28:00 on dist-test-slave-j0hf"
server_key: "9c6dbc187abf503167031180c04fdf71"
server_key_iv: "2218ac92f9c0edd6bfc668249c0c87db"
server_key_version: "encryptionkey@0"
I20260927 03:28:00.494757 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:28:00.495492 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:00.495671 1159 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:28:00.495764 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "97886abebfed42ef9df9dc81cd4c45fb"
format_stamp: "Formatted at 2026-09-27 03:28:00 on dist-test-slave-j0hf"
server_key: "9c6dbc187abf503167031180c04fdf71"
server_key_iv: "2218ac92f9c0edd6bfc668249c0c87db"
server_key_version: "encryptionkey@0"
I20260927 03:28:00.495836 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:00.503721 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:00.506063 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:00.506176 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:00.506347 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:00.506660 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:00.506695 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:00.506716 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:00.506762 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:00.512636 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34089
I20260927 03:28:00.512708 1287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34089 every 8 connection(s)
I20260927 03:28:00.513032 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:28:00 dist-test-slave-j0hf krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:00.519587 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 1159
I20260927 03:28:00.519690 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:28:00.519878 392 external_mini_cluster.cc:1428] Setting key b647963250957a1b4d293baaea65f55b
I20260927 03:28:00.523109 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:00.513815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59971 (local address 127.0.98.62:44809)
0927 03:28:00.513869 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:00.513872 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:00.514060 (+ 188us) server_negotiation.cc:407] Connection header received
0927 03:28:00.514716 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:00.514718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:00.514732 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:00.514786 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:00.515867 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.516374 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.517386 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.517592 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.517678 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:00.519905 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:00.519907 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:00.519907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:00.519921 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:00.521659 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:00.522086 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:00.522087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:00.522087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:00.522152 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:00.522429 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:00.522430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:00.522431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:00.522557 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:00.522634 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:00.522929 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 03:28:00.523019 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:28:00.523295 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:00.513771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44809 (local address 127.0.98.2:59971)
0927 03:28:00.513915 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:00.513974 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:00.514596 (+ 622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:00.514858 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:00.514862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:00.515575 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:00.515768 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:00.515773 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.516472 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.516473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:00.517290 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:00.517292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.517686 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.517687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:00.517820 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:00.518297 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:00.518305 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:00.519799 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:00.521757 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:00.521764 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:00.521766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:00.522005 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:00.522221 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:00.522222 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:00.522222 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:00.522342 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:00.522625 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:00.522631 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:00.522779 (+ 148us) client_negotiation.cc:769] Sending connection context
0927 03:28:00.522885 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 03:28:00.523048 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
I20260927 03:28:00.523866 1288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44809
I20260927 03:28:00.523969 1288 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:00.524178 1288 heartbeater.cc:511] Master 127.0.98.62:44809 requested a full tablet report, sending...
I20260927 03:28:00.524580 963 ts_manager.cc:194] Registered new tserver with Master: 97886abebfed42ef9df9dc81cd4c45fb (127.0.98.2:34089)
I20260927 03:28:00.525135 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59971
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:28:00.546900 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44809
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:00.630009 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:00.630227 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:00.630254 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:00.631582 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:00.631640 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:00.631671 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:28:00.633143 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44809
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:00.633404 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:00.633654 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:00.638747 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:00.638756 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:00.638761 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:00.639081 1293 server_base.cc:1034] running on GCE node
I20260927 03:28:00.639257 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:00.639549 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:00.640691 1293 hybrid_clock.cc:648] HybridClock initialized: now 1790479680640678 us; error 31 us; skew 500 ppm
Sep 27 03:28:00 dist-test-slave-j0hf krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:00.643291 1293 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:28:00.643837 1293 webserver.cc:460] Webserver started at http://127.0.98.3:42265/ using document root <none> and password file <none>
I20260927 03:28:00.644069 1293 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:00.644146 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:00.644289 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:00.645447 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-2/data/instance:
uuid: "212c3c9ba4d14cb0a7464e1adaa8733f"
format_stamp: "Formatted at 2026-09-27 03:28:00 on dist-test-slave-j0hf"
server_key: "957dffbdbb0541731584546a0779224d"
server_key_iv: "b1086cd10ff67b618ffc246984c6996f"
server_key_version: "encryptionkey@0"
I20260927 03:28:00.645828 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-2/wal/instance:
uuid: "212c3c9ba4d14cb0a7464e1adaa8733f"
format_stamp: "Formatted at 2026-09-27 03:28:00 on dist-test-slave-j0hf"
server_key: "957dffbdbb0541731584546a0779224d"
server_key_iv: "b1086cd10ff67b618ffc246984c6996f"
server_key_version: "encryptionkey@0"
I20260927 03:28:00.647235 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:28:00.648068 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:00.648262 1293 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:28:00.648351 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "212c3c9ba4d14cb0a7464e1adaa8733f"
format_stamp: "Formatted at 2026-09-27 03:28:00 on dist-test-slave-j0hf"
server_key: "957dffbdbb0541731584546a0779224d"
server_key_iv: "b1086cd10ff67b618ffc246984c6996f"
server_key_version: "encryptionkey@0"
I20260927 03:28:00.648424 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:00.672595 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:00.674849 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:00.675014 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:00.675282 1293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:00.675679 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:00.675740 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:00.675782 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:00.675810 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:00.682089 1293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37825
I20260927 03:28:00.682139 1421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37825 every 8 connection(s)
I20260927 03:28:00.682492 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:28:00 dist-test-slave-j0hf krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:00.692240 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 1293
I20260927 03:28:00.692345 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790479675865103-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:28:00.692517 392 external_mini_cluster.cc:1428] Setting key bf57d597912f6b593fae7e402d530867
I20260927 03:28:00.693106 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:00.683325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44799 (local address 127.0.98.62:44809)
0927 03:28:00.683393 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:00.683395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:00.683644 (+ 249us) server_negotiation.cc:407] Connection header received
0927 03:28:00.684113 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:00.684114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:00.684123 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:00.684183 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:00.685201 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.685720 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.687010 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.687219 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.687285 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:00.689757 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:00.689759 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:00.689759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:00.689766 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:00.691275 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:00.691893 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:00.691894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:00.691894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:00.691950 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:00.692303 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:00.692305 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:00.692305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:00.692464 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:00.692532 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:00.692915 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 03:28:00.693016 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:28:00.693298 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:00.683289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44809 (local address 127.0.98.3:44799)
0927 03:28:00.683503 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:00.683548 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:00.684032 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:00.684249 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:00.684252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:00.684838 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:00.685048 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:00.685054 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.685842 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.685844 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:00.686905 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:00.686909 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.687316 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.687317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:00.687548 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:00.688142 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:00.688152 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:00.689661 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:00.691411 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:00.691484 (+ 73us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:00.691487 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:00.691814 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:00.692051 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:00.692084 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:00.692085 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:00.692231 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:00.692543 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:00.692549 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:00.692732 (+ 183us) client_negotiation.cc:769] Sending connection context
0927 03:28:00.692844 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 03:28:00.693031 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":96,"threads_started":1}
I20260927 03:28:00.693836 1422 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44809
I20260927 03:28:00.693938 1422 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:00.694160 1422 heartbeater.cc:511] Master 127.0.98.62:44809 requested a full tablet report, sending...
I20260927 03:28:00.694578 963 ts_manager.cc:194] Registered new tserver with Master: 212c3c9ba4d14cb0a7464e1adaa8733f (127.0.98.3:37825)
I20260927 03:28:00.694967 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44799
I20260927 03:28:00.704380 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:28:00.712924 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:00.706094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42060 (local address 127.0.98.62:44809)
0927 03:28:00.706174 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:00.706176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:00.706255 (+ 79us) server_negotiation.cc:407] Connection header received
0927 03:28:00.706361 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:00.706361 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:00.706372 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:00.706481 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:00.707065 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.707760 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.708704 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.708912 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.709028 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:00.710276 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:00.710278 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:00.710278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:00.710287 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:00.711605 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:00.711999 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:00.712000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:00.712001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:00.712071 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:00.712285 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:00.712287 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:00.712287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:00.712468 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:00.712585 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:00.712711 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 03:28:00.712805 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 03:28:00.715539 963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:28:00.717206 963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:28:00.724232 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:00.719980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43118 (local address 127.0.98.3:37825)
0927 03:28:00.720267 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:00.720271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:00.720280 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:28:00.720316 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:00.720318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:00.720331 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:00.720418 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:00.720625 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.721696 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.723048 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.723827 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.723908 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:00.723967 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:00.724089 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 03:28:00.724138 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":85,"threads_started":1}
I20260927 03:28:00.724228 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:00.719771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37825 (local address 127.0.0.1:43118)
0927 03:28:00.720149 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:00.720161 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:00.720211 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:00.720421 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:00.720422 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:00.720423 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:00.720463 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:00.720464 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.721812 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.721812 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:00.722955 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:00.722957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.723907 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.723908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:00.723975 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:00.723987 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 03:28:00.724064 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 03:28:00.724135 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":64,"thread_start_us":82,"threads_started":1}
I20260927 03:28:00.724716 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:00.719780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37813 (local address 127.0.0.1:58312)
0927 03:28:00.720391 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:00.720401 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:00.720467 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:00.720758 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:00.720758 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:00.720760 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:00.720807 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:00.720808 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.721769 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.721769 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:00.723151 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:00.723153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.724150 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.724151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:00.724207 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:00.724211 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:28:00.724262 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 03:28:00.724650 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":83,"thread_start_us":93,"threads_started":1}
I20260927 03:28:00.725180 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:00.720018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58312 (local address 127.0.98.1:37813)
0927 03:28:00.720186 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:00.720189 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:00.720551 (+ 362us) server_negotiation.cc:407] Connection header received
0927 03:28:00.720595 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:00.720598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:00.720615 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:00.720681 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:00.720877 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.721571 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.723244 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.724058 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.724128 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:00.724179 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:00.725054 (+ 875us) server_negotiation.cc:299] Negotiation successful
0927 03:28:00.725105 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":73,"threads_started":1}
I20260927 03:28:00.725109 1356 tablet_service.cc:1467] Processing CreateTablet for tablet fc87d44b2fb94b2191e6850aa032db02 (DEFAULT_TABLE table=test-table [id=e60def89789c4887a3d78bb8ac33e84c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:00.725359 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:00.719762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34089 (local address 127.0.0.1:35274)
0927 03:28:00.719918 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:00.719960 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:00.720049 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:00.720327 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:00.720330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:00.720332 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:00.720426 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:00.720427 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.721930 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.721931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:00.723151 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:00.723153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.725130 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.725131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:00.725177 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:00.725181 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:28:00.725252 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 03:28:00.725294 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":33,"thread_start_us":48,"threads_started":1}
I20260927 03:28:00.725396 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:00.719888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35274 (local address 127.0.98.2:34089)
0927 03:28:00.720107 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:00.720112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:00.720124 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:28:00.720169 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:00.720171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:00.720189 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:00.720288 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:00.720534 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.721831 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.724218 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.724974 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.725042 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:00.725095 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:00.725267 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 03:28:00.725317 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":78,"threads_started":1}
I20260927 03:28:00.725514 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc87d44b2fb94b2191e6850aa032db02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:00.726039 1088 tablet_service.cc:1467] Processing CreateTablet for tablet fc87d44b2fb94b2191e6850aa032db02 (DEFAULT_TABLE table=test-table [id=e60def89789c4887a3d78bb8ac33e84c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:00.726039 1222 tablet_service.cc:1467] Processing CreateTablet for tablet fc87d44b2fb94b2191e6850aa032db02 (DEFAULT_TABLE table=test-table [id=e60def89789c4887a3d78bb8ac33e84c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:00.726374 1088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc87d44b2fb94b2191e6850aa032db02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:00.726390 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc87d44b2fb94b2191e6850aa032db02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:00.728515 1439 tablet_bootstrap.cc:492] T fc87d44b2fb94b2191e6850aa032db02 P 212c3c9ba4d14cb0a7464e1adaa8733f: Bootstrap starting.
I20260927 03:28:00.729172 1439 tablet_bootstrap.cc:654] T fc87d44b2fb94b2191e6850aa032db02 P 212c3c9ba4d14cb0a7464e1adaa8733f: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:00.729770 1440 tablet_bootstrap.cc:492] T fc87d44b2fb94b2191e6850aa032db02 P 97886abebfed42ef9df9dc81cd4c45fb: Bootstrap starting.
I20260927 03:28:00.729771 1439 log.cc:824] T fc87d44b2fb94b2191e6850aa032db02 P 212c3c9ba4d14cb0a7464e1adaa8733f: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:00.730506 1439 tablet_bootstrap.cc:492] T fc87d44b2fb94b2191e6850aa032db02 P 212c3c9ba4d14cb0a7464e1adaa8733f: No bootstrap required, opened a new log
I20260927 03:28:00.730602 1439 ts_tablet_manager.cc:1397] T fc87d44b2fb94b2191e6850aa032db02 P 212c3c9ba4d14cb0a7464e1adaa8733f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:28:00.730957 1440 tablet_bootstrap.cc:654] T fc87d44b2fb94b2191e6850aa032db02 P 97886abebfed42ef9df9dc81cd4c45fb: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:00.731474 1440 log.cc:824] T fc87d44b2fb94b2191e6850aa032db02 P 97886abebfed42ef9df9dc81cd4c45fb: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:00.731617 1441 tablet_bootstrap.cc:492] T fc87d44b2fb94b2191e6850aa032db02 P 3e1c5443fed74fdf8917a4dfdb0dd85b: Bootstrap starting.
I20260927 03:28:00.732183 1441 tablet_bootstrap.cc:654] T fc87d44b2fb94b2191e6850aa032db02 P 3e1c5443fed74fdf8917a4dfdb0dd85b: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:00.732343 1440 tablet_bootstrap.cc:492] T fc87d44b2fb94b2191e6850aa032db02 P 97886abebfed42ef9df9dc81cd4c45fb: No bootstrap required, opened a new log
I20260927 03:28:00.732422 1440 ts_tablet_manager.cc:1397] T fc87d44b2fb94b2191e6850aa032db02 P 97886abebfed42ef9df9dc81cd4c45fb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 03:28:00.732519 1439 raft_consensus.cc:348] T fc87d44b2fb94b2191e6850aa032db02 P 212c3c9ba4d14cb0a7464e1adaa8733f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97886abebfed42ef9df9dc81cd4c45fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34089 } } peers { permanent_uuid: "3e1c5443fed74fdf8917a4dfdb0dd85b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "212c3c9ba4d14cb0a7464e1adaa8733f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37825 } }
I20260927 03:28:00.732697 1439 raft_consensus.cc:374] T fc87d44b2fb94b2191e6850aa032db02 P 212c3c9ba4d14cb0a7464e1adaa8733f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:00.732740 1439 raft_consensus.cc:729] T fc87d44b2fb94b2191e6850aa032db02 P 212c3c9ba4d14cb0a7464e1adaa8733f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212c3c9ba4d14cb0a7464e1adaa8733f, State: Initialized, Role: FOLLOWER
I20260927 03:28:00.732736 1441 log.cc:824] T fc87d44b2fb94b2191e6850aa032db02 P 3e1c5443fed74fdf8917a4dfdb0dd85b: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:00.732853 1439 consensus_queue.cc:260] T fc87d44b2fb94b2191e6850aa032db02 P 212c3c9ba4d14cb0a7464e1adaa8733f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97886abebfed42ef9df9dc81cd4c45fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34089 } } peers { permanent_uuid: "3e1c5443fed74fdf8917a4dfdb0dd85b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "212c3c9ba4d14cb0a7464e1adaa8733f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37825 } }
I20260927 03:28:00.733152 1439 ts_tablet_manager.cc:1428] T fc87d44b2fb94b2191e6850aa032db02 P 212c3c9ba4d14cb0a7464e1adaa8733f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:28:00.733279 1422 heartbeater.cc:503] Master 127.0.98.62:44809 was elected leader, sending a full tablet report...
I20260927 03:28:00.734368 1440 raft_consensus.cc:348] T fc87d44b2fb94b2191e6850aa032db02 P 97886abebfed42ef9df9dc81cd4c45fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97886abebfed42ef9df9dc81cd4c45fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34089 } } peers { permanent_uuid: "3e1c5443fed74fdf8917a4dfdb0dd85b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "212c3c9ba4d14cb0a7464e1adaa8733f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37825 } }
I20260927 03:28:00.734498 1440 raft_consensus.cc:374] T fc87d44b2fb94b2191e6850aa032db02 P 97886abebfed42ef9df9dc81cd4c45fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:00.734519 1440 raft_consensus.cc:729] T fc87d44b2fb94b2191e6850aa032db02 P 97886abebfed42ef9df9dc81cd4c45fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97886abebfed42ef9df9dc81cd4c45fb, State: Initialized, Role: FOLLOWER
I20260927 03:28:00.734664 1440 consensus_queue.cc:260] T fc87d44b2fb94b2191e6850aa032db02 P 97886abebfed42ef9df9dc81cd4c45fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97886abebfed42ef9df9dc81cd4c45fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34089 } } peers { permanent_uuid: "3e1c5443fed74fdf8917a4dfdb0dd85b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "212c3c9ba4d14cb0a7464e1adaa8733f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37825 } }
I20260927 03:28:00.734921 1440 ts_tablet_manager.cc:1428] T fc87d44b2fb94b2191e6850aa032db02 P 97886abebfed42ef9df9dc81cd4c45fb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 03:28:00.734951 1288 heartbeater.cc:503] Master 127.0.98.62:44809 was elected leader, sending a full tablet report...
I20260927 03:28:00.735528 1441 tablet_bootstrap.cc:492] T fc87d44b2fb94b2191e6850aa032db02 P 3e1c5443fed74fdf8917a4dfdb0dd85b: No bootstrap required, opened a new log
I20260927 03:28:00.735591 1441 ts_tablet_manager.cc:1397] T fc87d44b2fb94b2191e6850aa032db02 P 3e1c5443fed74fdf8917a4dfdb0dd85b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 03:28:00.737313 1441 raft_consensus.cc:348] T fc87d44b2fb94b2191e6850aa032db02 P 3e1c5443fed74fdf8917a4dfdb0dd85b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97886abebfed42ef9df9dc81cd4c45fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34089 } } peers { permanent_uuid: "3e1c5443fed74fdf8917a4dfdb0dd85b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "212c3c9ba4d14cb0a7464e1adaa8733f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37825 } }
I20260927 03:28:00.737454 1441 raft_consensus.cc:374] T fc87d44b2fb94b2191e6850aa032db02 P 3e1c5443fed74fdf8917a4dfdb0dd85b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:00.737476 1441 raft_consensus.cc:729] T fc87d44b2fb94b2191e6850aa032db02 P 3e1c5443fed74fdf8917a4dfdb0dd85b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e1c5443fed74fdf8917a4dfdb0dd85b, State: Initialized, Role: FOLLOWER
I20260927 03:28:00.737600 1441 consensus_queue.cc:260] T fc87d44b2fb94b2191e6850aa032db02 P 3e1c5443fed74fdf8917a4dfdb0dd85b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97886abebfed42ef9df9dc81cd4c45fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34089 } } peers { permanent_uuid: "3e1c5443fed74fdf8917a4dfdb0dd85b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "212c3c9ba4d14cb0a7464e1adaa8733f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37825 } }
I20260927 03:28:00.737917 1441 ts_tablet_manager.cc:1428] T fc87d44b2fb94b2191e6850aa032db02 P 3e1c5443fed74fdf8917a4dfdb0dd85b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:28:00.737948 1154 heartbeater.cc:503] Master 127.0.98.62:44809 was elected leader, sending a full tablet report...
W20260927 03:28:00.763645 1289 tablet.cc:2369] T fc87d44b2fb94b2191e6850aa032db02 P 97886abebfed42ef9df9dc81cd4c45fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:28:00.810230 1446 raft_consensus.cc:482] T fc87d44b2fb94b2191e6850aa032db02 P 97886abebfed42ef9df9dc81cd4c45fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:28:00.810424 1446 raft_consensus.cc:504] T fc87d44b2fb94b2191e6850aa032db02 P 97886abebfed42ef9df9dc81cd4c45fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97886abebfed42ef9df9dc81cd4c45fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34089 } } peers { permanent_uuid: "3e1c5443fed74fdf8917a4dfdb0dd85b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "212c3c9ba4d14cb0a7464e1adaa8733f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37825 } }
I20260927 03:28:00.810840 1446 leader_election.cc:288] T fc87d44b2fb94b2191e6850aa032db02 P 97886abebfed42ef9df9dc81cd4c45fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e1c5443fed74fdf8917a4dfdb0dd85b (127.0.98.1:37813), 212c3c9ba4d14cb0a7464e1adaa8733f (127.0.98.3:37825)
I20260927 03:28:00.814865 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:00.811180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39317 (local address 127.0.98.1:37813)
0927 03:28:00.811230 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:00.811232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:00.811399 (+ 167us) server_negotiation.cc:407] Connection header received
0927 03:28:00.811458 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:00.811459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:00.811471 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:00.811550 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:00.811691 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.812367 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.813850 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.814454 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.814525 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:00.814537 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:00.814745 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 03:28:00.814789 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 03:28:00.814905 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:00.811046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37825 (local address 127.0.98.2:34049)
0927 03:28:00.811118 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:00.811132 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:00.811201 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:00.811370 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:00.811371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:00.811373 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:00.811473 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:00.811477 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.812442 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.812442 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:00.813759 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:00.813762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.814620 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.814623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:00.814688 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:00.814691 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:28:00.814764 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 03:28:00.814803 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":38}
I20260927 03:28:00.814929 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:00.811046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37813 (local address 127.0.98.2:39317)
0927 03:28:00.811299 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:00.811308 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:00.811384 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:00.811562 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:00.811562 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:00.811564 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:00.811621 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:00.811623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.812442 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.812442 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:00.813759 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:00.813762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.814585 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.814586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:00.814670 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:00.814673 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:28:00.814715 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 03:28:00.814776 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":117,"thread_start_us":106,"threads_started":1}
I20260927 03:28:00.814917 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:00.811184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34049 (local address 127.0.98.3:37825)
0927 03:28:00.811230 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:00.811232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:00.811243 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:28:00.811272 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:00.811273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:00.811290 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:00.811343 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:00.811554 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.812351 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.813851 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.814497 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.814591 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:00.814600 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:00.814807 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 03:28:00.814850 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:28:00.815237 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc87d44b2fb94b2191e6850aa032db02" candidate_uuid: "97886abebfed42ef9df9dc81cd4c45fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212c3c9ba4d14cb0a7464e1adaa8733f" is_pre_election: true
I20260927 03:28:00.815253 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc87d44b2fb94b2191e6850aa032db02" candidate_uuid: "97886abebfed42ef9df9dc81cd4c45fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e1c5443fed74fdf8917a4dfdb0dd85b" is_pre_election: true
I20260927 03:28:00.815400 1376 raft_consensus.cc:2454] T fc87d44b2fb94b2191e6850aa032db02 P 212c3c9ba4d14cb0a7464e1adaa8733f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97886abebfed42ef9df9dc81cd4c45fb in term 0.
I20260927 03:28:00.815400 1108 raft_consensus.cc:2454] T fc87d44b2fb94b2191e6850aa032db02 P 3e1c5443fed74fdf8917a4dfdb0dd85b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97886abebfed42ef9df9dc81cd4c45fb in term 0.
I20260927 03:28:00.815693 1178 leader_election.cc:302] T fc87d44b2fb94b2191e6850aa032db02 P 97886abebfed42ef9df9dc81cd4c45fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e1c5443fed74fdf8917a4dfdb0dd85b, 97886abebfed42ef9df9dc81cd4c45fb; no voters:
I20260927 03:28:00.815876 1446 raft_consensus.cc:2781] T fc87d44b2fb94b2191e6850aa032db02 P 97886abebfed42ef9df9dc81cd4c45fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:28:00.815930 1446 raft_consensus.cc:482] T fc87d44b2fb94b2191e6850aa032db02 P 97886abebfed42ef9df9dc81cd4c45fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:28:00.815958 1446 raft_consensus.cc:3037] T fc87d44b2fb94b2191e6850aa032db02 P 97886abebfed42ef9df9dc81cd4c45fb [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:00.816632 1446 raft_consensus.cc:504] T fc87d44b2fb94b2191e6850aa032db02 P 97886abebfed42ef9df9dc81cd4c45fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97886abebfed42ef9df9dc81cd4c45fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34089 } } peers { permanent_uuid: "3e1c5443fed74fdf8917a4dfdb0dd85b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "212c3c9ba4d14cb0a7464e1adaa8733f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37825 } }
I20260927 03:28:00.816778 1446 leader_election.cc:288] T fc87d44b2fb94b2191e6850aa032db02 P 97886abebfed42ef9df9dc81cd4c45fb [CANDIDATE]: Term 1 election: Requested vote from peers 3e1c5443fed74fdf8917a4dfdb0dd85b (127.0.98.1:37813), 212c3c9ba4d14cb0a7464e1adaa8733f (127.0.98.3:37825)
I20260927 03:28:00.816908 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc87d44b2fb94b2191e6850aa032db02" candidate_uuid: "97886abebfed42ef9df9dc81cd4c45fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e1c5443fed74fdf8917a4dfdb0dd85b"
I20260927 03:28:00.816932 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc87d44b2fb94b2191e6850aa032db02" candidate_uuid: "97886abebfed42ef9df9dc81cd4c45fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212c3c9ba4d14cb0a7464e1adaa8733f"
I20260927 03:28:00.816993 1376 raft_consensus.cc:3037] T fc87d44b2fb94b2191e6850aa032db02 P 212c3c9ba4d14cb0a7464e1adaa8733f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:00.816987 1108 raft_consensus.cc:3037] T fc87d44b2fb94b2191e6850aa032db02 P 3e1c5443fed74fdf8917a4dfdb0dd85b [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:00.817883 1376 raft_consensus.cc:2454] T fc87d44b2fb94b2191e6850aa032db02 P 212c3c9ba4d14cb0a7464e1adaa8733f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97886abebfed42ef9df9dc81cd4c45fb in term 1.
I20260927 03:28:00.817883 1108 raft_consensus.cc:2454] T fc87d44b2fb94b2191e6850aa032db02 P 3e1c5443fed74fdf8917a4dfdb0dd85b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97886abebfed42ef9df9dc81cd4c45fb in term 1.
I20260927 03:28:00.818048 1178 leader_election.cc:302] T fc87d44b2fb94b2191e6850aa032db02 P 97886abebfed42ef9df9dc81cd4c45fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e1c5443fed74fdf8917a4dfdb0dd85b, 97886abebfed42ef9df9dc81cd4c45fb; no voters:
I20260927 03:28:00.818171 1446 raft_consensus.cc:2781] T fc87d44b2fb94b2191e6850aa032db02 P 97886abebfed42ef9df9dc81cd4c45fb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:00.818322 1446 raft_consensus.cc:686] T fc87d44b2fb94b2191e6850aa032db02 P 97886abebfed42ef9df9dc81cd4c45fb [term 1 LEADER]: Becoming Leader. State: Replica: 97886abebfed42ef9df9dc81cd4c45fb, State: Running, Role: LEADER
I20260927 03:28:00.818423 1446 consensus_queue.cc:237] T fc87d44b2fb94b2191e6850aa032db02 P 97886abebfed42ef9df9dc81cd4c45fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97886abebfed42ef9df9dc81cd4c45fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34089 } } peers { permanent_uuid: "3e1c5443fed74fdf8917a4dfdb0dd85b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } } peers { permanent_uuid: "212c3c9ba4d14cb0a7464e1adaa8733f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37825 } }
I20260927 03:28:00.819245 962 catalog_manager.cc:5495] T fc87d44b2fb94b2191e6850aa032db02 P 97886abebfed42ef9df9dc81cd4c45fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 97886abebfed42ef9df9dc81cd4c45fb (127.0.98.2). New cstate: current_term: 1 leader_uuid: "97886abebfed42ef9df9dc81cd4c45fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97886abebfed42ef9df9dc81cd4c45fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e1c5443fed74fdf8917a4dfdb0dd85b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212c3c9ba4d14cb0a7464e1adaa8733f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37825 } health_report { overall_health: UNKNOWN } } }
Sep 27 03:28:00 dist-test-slave-j0hf krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 03:28:00.847028 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:00.838850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58324 (local address 127.0.98.1:37813)
0927 03:28:00.838948 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:00.838950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:00.839037 (+ 87us) server_negotiation.cc:407] Connection header received
0927 03:28:00.839147 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:00.839148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:00.839159 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:00.839217 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:00.839969 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.840516 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.841318 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.841552 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.841640 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:00.844356 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:00.844369 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:00.844370 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:00.844380 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:00.845929 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:00.846304 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:00.846307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:00.846308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:00.846367 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:00.846556 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:00.846557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:00.846557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:00.846703 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:00.846821 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:00.846889 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 03:28:00.846957 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 03:28:00 dist-test-slave-j0hf krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479680, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260927 03:28:00.859114 1155 tablet.cc:2369] T fc87d44b2fb94b2191e6850aa032db02 P 3e1c5443fed74fdf8917a4dfdb0dd85b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 03:28:00 dist-test-slave-j0hf krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479680, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 03:28:00.870425 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:00.862829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58328 (local address 127.0.98.1:37813)
0927 03:28:00.862931 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:00.862933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:00.863040 (+ 107us) server_negotiation.cc:407] Connection header received
0927 03:28:00.863143 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:00.863144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:00.863154 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:00.863208 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:00.863915 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.864397 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.865161 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.865356 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.865445 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:00.867820 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:00.867821 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:00.867822 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:00.867830 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:00.869119 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:00.869552 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:00.869554 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:00.869554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:00.869615 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:00.869824 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:00.869825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:00.869826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:00.869943 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:00.870051 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:00.870239 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 03:28:00.870321 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 03:28:00 dist-test-slave-j0hf krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 03:28:00 dist-test-slave-j0hf krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 03:28:00.892884 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:00.886010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58336 (local address 127.0.98.1:37813)
0927 03:28:00.886081 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:00.886083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:00.886091 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:28:00.886116 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:00.886116 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:00.886126 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:00.886178 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:00.886736 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.887216 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.888009 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:00.888204 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:00.888311 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:00.890481 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:00.890483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:00.890483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:00.890491 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:00.891757 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:00.892104 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:00.892106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:00.892106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:00.892162 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:00.892395 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:00.892396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:00.892396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:00.892505 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:00.892589 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:00.892751 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 03:28:00.892814 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 03:28:00.893356 1068 tablet_service.cc:2908] Rejecting scan request for tablet fc87d44b2fb94b2191e6850aa032db02: Uninitialized: clean time has not yet been initialized
I20260927 03:28:00.894351 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 1025
I20260927 03:28:00.899608 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 1159
I20260927 03:28:00.904186 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 1293
I20260927 03:28:00.909042 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 931
2026-09-27T03:28:00Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2098 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:28:00 dist-test-slave-j0hf krb5kdc[1463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:28:00 dist-test-slave-j0hf krb5kdc[1463](info): set up 2 sockets
Sep 27 03:28:00 dist-test-slave-j0hf krb5kdc[1463](info): commencing operation
krb5kdc: starting...
W20260927 03:28:01.967149 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:28:02 dist-test-slave-j0hf krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:28:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:28:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:28:02.090035 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40781
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:02.175314 1476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:02.175568 1476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:02.175593 1476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:02.176740 1476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:28:02.176780 1476 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 03:28:02.176793 1476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:02.176806 1476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:02.176819 1476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:28:02.176831 1476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:28:02.178316 1476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40781
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:02.178542 1476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:02.178733 1476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:02.183773 1484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:02.183835 1482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:02.184012 1476 server_base.cc:1034] running on GCE node
W20260927 03:28:02.183782 1481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:02.184350 1476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:02.184676 1476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:02.185922 1476 hybrid_clock.cc:648] HybridClock initialized: now 1790479682185884 us; error 44 us; skew 500 ppm
Sep 27 03:28:02 dist-test-slave-j0hf krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:02.188706 1476 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:28:02.189253 1476 webserver.cc:460] Webserver started at http://127.0.98.62:45665/ using document root <none> and password file <none>
I20260927 03:28:02.189509 1476 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:02.189580 1476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:02.189744 1476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:02.190829 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "cc115db5035545bdb6cc4d97816cf042"
format_stamp: "Formatted at 2026-09-27 03:28:02 on dist-test-slave-j0hf"
server_key: "6bdbadca5af66b371f486b24f741ba64"
server_key_iv: "4393d8a0d1efeaf0f4ac91a91a39c18e"
server_key_version: "encryptionkey@0"
I20260927 03:28:02.191220 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "cc115db5035545bdb6cc4d97816cf042"
format_stamp: "Formatted at 2026-09-27 03:28:02 on dist-test-slave-j0hf"
server_key: "6bdbadca5af66b371f486b24f741ba64"
server_key_iv: "4393d8a0d1efeaf0f4ac91a91a39c18e"
server_key_version: "encryptionkey@0"
I20260927 03:28:02.192703 1476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:02.193441 1491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:02.193638 1476 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:02.193748 1476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "cc115db5035545bdb6cc4d97816cf042"
format_stamp: "Formatted at 2026-09-27 03:28:02 on dist-test-slave-j0hf"
server_key: "6bdbadca5af66b371f486b24f741ba64"
server_key_iv: "4393d8a0d1efeaf0f4ac91a91a39c18e"
server_key_version: "encryptionkey@0"
I20260927 03:28:02.193827 1476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:02.206264 1476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:02.208397 1476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:02.208573 1476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:02.213418 1476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40781
I20260927 03:28:02.213454 1553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40781 every 8 connection(s)
I20260927 03:28:02.213821 1476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:28:02.214561 1554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:02.215128 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 1476
I20260927 03:28:02.215234 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:28:02.215457 392 external_mini_cluster.cc:1428] Setting key 41f187e070dc411d3562410edd6b904e
I20260927 03:28:02.217494 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc115db5035545bdb6cc4d97816cf042: Bootstrap starting.
I20260927 03:28:02.218155 1554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc115db5035545bdb6cc4d97816cf042: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:02.218478 1554 log.cc:824] T 00000000000000000000000000000000 P cc115db5035545bdb6cc4d97816cf042: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:02.219214 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc115db5035545bdb6cc4d97816cf042: No bootstrap required, opened a new log
I20260927 03:28:02.221423 1554 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc115db5035545bdb6cc4d97816cf042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc115db5035545bdb6cc4d97816cf042" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40781 } }
I20260927 03:28:02.221598 1554 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc115db5035545bdb6cc4d97816cf042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:02.221652 1554 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc115db5035545bdb6cc4d97816cf042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc115db5035545bdb6cc4d97816cf042, State: Initialized, Role: FOLLOWER
I20260927 03:28:02.221828 1554 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc115db5035545bdb6cc4d97816cf042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc115db5035545bdb6cc4d97816cf042" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40781 } }
I20260927 03:28:02.221936 1554 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc115db5035545bdb6cc4d97816cf042 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:28:02.221985 1554 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc115db5035545bdb6cc4d97816cf042 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:28:02.222051 1554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc115db5035545bdb6cc4d97816cf042 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 03:28:02 dist-test-slave-j0hf krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:02.222693 1554 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc115db5035545bdb6cc4d97816cf042 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc115db5035545bdb6cc4d97816cf042" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40781 } }
I20260927 03:28:02.222820 1554 leader_election.cc:302] T 00000000000000000000000000000000 P cc115db5035545bdb6cc4d97816cf042 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc115db5035545bdb6cc4d97816cf042; no voters:
I20260927 03:28:02.223040 1554 leader_election.cc:288] T 00000000000000000000000000000000 P cc115db5035545bdb6cc4d97816cf042 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:28:02.223070 1559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc115db5035545bdb6cc4d97816cf042 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:02.223201 1559 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc115db5035545bdb6cc4d97816cf042 [term 1 LEADER]: Becoming Leader. State: Replica: cc115db5035545bdb6cc4d97816cf042, State: Running, Role: LEADER
I20260927 03:28:02.223336 1554 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc115db5035545bdb6cc4d97816cf042 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:28:02.223362 1559 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc115db5035545bdb6cc4d97816cf042 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc115db5035545bdb6cc4d97816cf042" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40781 } }
I20260927 03:28:02.223775 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc115db5035545bdb6cc4d97816cf042 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc115db5035545bdb6cc4d97816cf042. Latest consensus state: current_term: 1 leader_uuid: "cc115db5035545bdb6cc4d97816cf042" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc115db5035545bdb6cc4d97816cf042" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40781 } } }
I20260927 03:28:02.223789 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc115db5035545bdb6cc4d97816cf042 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc115db5035545bdb6cc4d97816cf042" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc115db5035545bdb6cc4d97816cf042" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40781 } } }
I20260927 03:28:02.223979 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc115db5035545bdb6cc4d97816cf042 [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:02.224007 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc115db5035545bdb6cc4d97816cf042 [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:02.224359 1568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:28:02.224946 1568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:28:02.227119 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:02.216048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49936 (local address 127.0.98.62:40781)
0927 03:28:02.216238 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:02.216246 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:02.216273 (+ 27us) server_negotiation.cc:407] Connection header received
0927 03:28:02.216671 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:02.216678 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:02.216725 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:02.216883 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:02.217522 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.218571 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.219323 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.219841 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.219977 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:02.222832 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:02.222836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:02.222837 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:02.222850 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:02.225226 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:02.225675 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:02.225678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:02.225679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:02.225824 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:02.226104 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:02.226106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:02.226106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:02.226438 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:02.226596 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:02.226740 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 03:28:02.226906 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
I20260927 03:28:02.227255 1568 catalog_manager.cc:1347] Generated new cluster ID: d9f0ef559cdc43978893802faa8db506
I20260927 03:28:02.227291 1568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:28:02.256839 1568 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:28:02.257431 1568 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:28:02.264674 1568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc115db5035545bdb6cc4d97816cf042: Generated new TSK 0
I20260927 03:28:02.264971 1568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:28:02.299631 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40781
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:02.384205 1580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:02.384464 1580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:02.384505 1580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:02.385704 1580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:02.385773 1580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:02.385812 1580 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:28:02.385871 1580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:28:02.387497 1580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:02.387797 1580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:02.388041 1580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:28:02.393364 1580 server_base.cc:1034] running on GCE node
W20260927 03:28:02.393290 1588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:02.393335 1585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:02.393318 1586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:02.393750 1580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:02.394052 1580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:02.395259 1580 hybrid_clock.cc:648] HybridClock initialized: now 1790479682395265 us; error 51 us; skew 500 ppm
Sep 27 03:28:02 dist-test-slave-j0hf krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:02.398203 1580 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:28:02.398691 1580 webserver.cc:460] Webserver started at http://127.0.98.1:43857/ using document root <none> and password file <none>
I20260927 03:28:02.398895 1580 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:02.398938 1580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:02.399027 1580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:02.400126 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-0/data/instance:
uuid: "6fadb3b6c4c04523806c27d488b1cec2"
format_stamp: "Formatted at 2026-09-27 03:28:02 on dist-test-slave-j0hf"
server_key: "b0ff4ceb56be58f03acf8abfa0581835"
server_key_iv: "430935c156ba2aa3c3d63ce5e4770716"
server_key_version: "encryptionkey@0"
I20260927 03:28:02.400444 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6fadb3b6c4c04523806c27d488b1cec2"
format_stamp: "Formatted at 2026-09-27 03:28:02 on dist-test-slave-j0hf"
server_key: "b0ff4ceb56be58f03acf8abfa0581835"
server_key_iv: "430935c156ba2aa3c3d63ce5e4770716"
server_key_version: "encryptionkey@0"
I20260927 03:28:02.401773 1580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:28:02.402541 1595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:02.402756 1580 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:02.402875 1580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "6fadb3b6c4c04523806c27d488b1cec2"
format_stamp: "Formatted at 2026-09-27 03:28:02 on dist-test-slave-j0hf"
server_key: "b0ff4ceb56be58f03acf8abfa0581835"
server_key_iv: "430935c156ba2aa3c3d63ce5e4770716"
server_key_version: "encryptionkey@0"
I20260927 03:28:02.402953 1580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:02.410010 1580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:02.412389 1580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:02.412561 1580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:02.413134 1580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:02.413194 1580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:02.413235 1580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:02.413256 1580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 03:28:02 dist-test-slave-j0hf krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:02.420881 1580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37057
I20260927 03:28:02.420964 1710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37057 every 8 connection(s)
I20260927 03:28:02.421320 1580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:28:02.423764 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:02.413570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41271 (local address 127.0.98.62:40781)
0927 03:28:02.413643 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:02.413645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:02.413889 (+ 244us) server_negotiation.cc:407] Connection header received
0927 03:28:02.414419 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:02.414421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:02.414435 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:02.414516 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:02.415290 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.415910 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.417195 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.417452 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.417551 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:02.420982 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:02.420984 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:02.420984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:02.420993 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:02.422428 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:02.422814 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:02.422815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:02.422815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:02.422865 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:02.423096 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:02.423096 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:02.423097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:02.423223 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:02.423315 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:02.423624 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 03:28:02.423698 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 03:28:02.423957 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:02.413581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40781 (local address 127.0.98.1:41271)
0927 03:28:02.413743 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:02.413782 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:02.414318 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:02.414633 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:02.414637 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:02.414968 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:02.415176 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:02.415181 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.416006 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.416008 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:02.417096 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:02.417099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.417562 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.417563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:02.417705 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:02.418336 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:02.418348 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:02.420855 (+ 2507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:02.422506 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:02.422511 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:02.422513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:02.422743 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:02.422922 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:02.422922 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:02.422923 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:02.423041 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:02.423304 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:02.423310 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:02.423481 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 03:28:02.423590 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 03:28:02.423720 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
I20260927 03:28:02.424505 1711 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40781
I20260927 03:28:02.424595 1711 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:02.424778 1711 heartbeater.cc:511] Master 127.0.98.62:40781 requested a full tablet report, sending...
I20260927 03:28:02.425137 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 1580
I20260927 03:28:02.425243 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:28:02.425434 392 external_mini_cluster.cc:1428] Setting key 9ad566c17c9472da10e5a0958a72321f
I20260927 03:28:02.425989 1507 ts_manager.cc:194] Registered new tserver with Master: 6fadb3b6c4c04523806c27d488b1cec2 (127.0.98.1:37057)
I20260927 03:28:02.426942 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41271
I20260927 03:28:02.432248 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:02.425633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55825 (local address 127.0.98.62:40781)
0927 03:28:02.425684 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:02.425685 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:02.425808 (+ 123us) server_negotiation.cc:407] Connection header received
0927 03:28:02.425876 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:02.425877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:02.425887 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:02.425940 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:02.426377 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.426862 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.427922 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.428131 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.428219 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:02.429286 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:02.429287 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:02.429288 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:02.429300 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:02.430951 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:02.431335 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:02.431336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:02.431337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:02.431399 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:02.431676 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:02.431677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:02.431677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:02.431807 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:02.431887 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:02.432073 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 03:28:02.432151 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 03:28:02.432264 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:02.425579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40781 (local address 127.0.98.1:55825)
0927 03:28:02.425758 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:02.425769 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:02.425822 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:02.426010 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:02.426011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:02.426224 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:02.426288 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:02.426290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.426955 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.426956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:02.427815 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:02.427817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.428219 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.428220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:02.428288 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:02.428780 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:02.428785 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:02.429142 (+ 357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:02.431049 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:02.431050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:02.431051 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:02.431247 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:02.431538 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:02.431539 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:02.431539 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:02.431607 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:02.431880 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:02.431881 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:02.431945 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 03:28:02.432042 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 03:28:02.432154 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:28:02.452445 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40781
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:02.535609 1720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:02.535831 1720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:02.535856 1720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:02.537068 1720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:02.537117 1720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:02.537138 1720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:28:02.537156 1720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:28:02.538728 1720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:02.539001 1720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:02.539265 1720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:02.544555 1728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:02.544749 1720 server_base.cc:1034] running on GCE node
W20260927 03:28:02.544562 1726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:02.544652 1725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:02.545063 1720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:02.545324 1720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:02.546516 1720 hybrid_clock.cc:648] HybridClock initialized: now 1790479682546496 us; error 48 us; skew 500 ppm
Sep 27 03:28:02 dist-test-slave-j0hf krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:02.549348 1720 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:28:02.549892 1720 webserver.cc:460] Webserver started at http://127.0.98.2:42905/ using document root <none> and password file <none>
I20260927 03:28:02.550132 1720 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:02.550197 1720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:02.550344 1720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:02.551506 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-1/data/instance:
uuid: "5803b8bb7755400e8d5c81a565fa588a"
format_stamp: "Formatted at 2026-09-27 03:28:02 on dist-test-slave-j0hf"
server_key: "0d82ece24954a11c763ac063929689d6"
server_key_iv: "4b2d10adfde0f42d74d57451446be289"
server_key_version: "encryptionkey@0"
I20260927 03:28:02.551887 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5803b8bb7755400e8d5c81a565fa588a"
format_stamp: "Formatted at 2026-09-27 03:28:02 on dist-test-slave-j0hf"
server_key: "0d82ece24954a11c763ac063929689d6"
server_key_iv: "4b2d10adfde0f42d74d57451446be289"
server_key_version: "encryptionkey@0"
I20260927 03:28:02.553279 1720 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:28:02.554086 1735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:02.554340 1720 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:28:02.554463 1720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "5803b8bb7755400e8d5c81a565fa588a"
format_stamp: "Formatted at 2026-09-27 03:28:02 on dist-test-slave-j0hf"
server_key: "0d82ece24954a11c763ac063929689d6"
server_key_iv: "4b2d10adfde0f42d74d57451446be289"
server_key_version: "encryptionkey@0"
I20260927 03:28:02.554556 1720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:02.566231 1720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:02.568921 1720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:02.569139 1720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:02.569885 1720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:02.569943 1720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:02.569989 1720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:02.570003 1720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 03:28:02 dist-test-slave-j0hf krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:02.577638 1720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37755
I20260927 03:28:02.577749 1850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37755 every 8 connection(s)
I20260927 03:28:02.578073 1720 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:28:02.580336 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:02.570446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37957 (local address 127.0.98.62:40781)
0927 03:28:02.570520 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:02.570523 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:02.570791 (+ 268us) server_negotiation.cc:407] Connection header received
0927 03:28:02.571345 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:02.571347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:02.571367 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:02.571502 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:02.572221 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.573070 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.574177 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.574527 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.574617 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:02.577411 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:02.577413 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:02.577413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:02.577423 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:02.579060 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:02.579493 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:02.579494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:02.579495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:02.579562 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:02.579792 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:02.579792 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:02.579793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:02.579916 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:02.579971 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:02.580206 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 03:28:02.580271 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:28:02.580648 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:02.570417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40781 (local address 127.0.98.2:37957)
0927 03:28:02.570635 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:02.570681 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:02.571240 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:02.571586 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:02.571589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:02.571886 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:02.572137 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:02.572142 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.573208 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.573209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:02.574087 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:02.574089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.574647 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.574648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:02.574788 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:02.575363 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:02.575372 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:02.577283 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:02.579135 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:02.579139 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:02.579141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:02.579355 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:02.579631 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:02.579632 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:02.579632 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:02.579732 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:02.579973 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:02.579977 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:02.580096 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 03:28:02.580180 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 03:28:02.580381 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":140,"threads_started":1}
I20260927 03:28:02.581157 1851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40781
I20260927 03:28:02.581269 1851 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:02.581485 1851 heartbeater.cc:511] Master 127.0.98.62:40781 requested a full tablet report, sending...
I20260927 03:28:02.581929 1508 ts_manager.cc:194] Registered new tserver with Master: 5803b8bb7755400e8d5c81a565fa588a (127.0.98.2:37755)
I20260927 03:28:02.582386 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37957
I20260927 03:28:02.587970 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:02.582240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36957 (local address 127.0.98.62:40781)
0927 03:28:02.582282 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:02.582284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:02.582407 (+ 123us) server_negotiation.cc:407] Connection header received
0927 03:28:02.582478 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:02.582478 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:02.582488 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:02.582537 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:02.582911 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.583562 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.584261 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.584431 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.584478 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:02.585268 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:02.585269 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:02.585269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:02.585275 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:02.586688 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:02.587148 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:02.587149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:02.587149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:02.587201 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:02.587349 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:02.587349 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:02.587349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:02.587578 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:02.587634 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:02.587845 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 03:28:02.587904 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:28:02.587966 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 1720
I20260927 03:28:02.588060 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:28:02.588279 392 external_mini_cluster.cc:1428] Setting key 27a8c6c8637e8b365c10ea49b8bca3fc
I20260927 03:28:02.588310 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:02.582183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40781 (local address 127.0.98.2:36957)
0927 03:28:02.582341 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:02.582349 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:02.582420 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:02.582548 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:02.582550 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:02.582760 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:02.582830 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:02.582832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.583653 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.583654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:02.584188 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:02.584190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.584474 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.584475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:02.584518 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:02.584912 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:02.584916 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:02.585196 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:02.586862 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:02.586864 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:02.586864 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:02.587068 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:02.587252 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:02.587253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:02.587253 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:02.587301 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:02.587648 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:02.587648 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:02.587694 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 03:28:02.587818 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 03:28:02.587940 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":76,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:28:02.614780 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40781
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:02.696406 1860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:02.696652 1860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:02.696695 1860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:02.698004 1860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:02.698078 1860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:02.698119 1860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:28:02.698176 1860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:28:02.699843 1860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:02.700133 1860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:02.700385 1860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:02.705581 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:02.705650 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:02.705638 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:02.705935 1860 server_base.cc:1034] running on GCE node
I20260927 03:28:02.706171 1860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:02.706436 1860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:02.707597 1860 hybrid_clock.cc:648] HybridClock initialized: now 1790479682707573 us; error 38 us; skew 500 ppm
Sep 27 03:28:02 dist-test-slave-j0hf krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:02.710157 1860 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:28:02.710672 1860 webserver.cc:460] Webserver started at http://127.0.98.3:46649/ using document root <none> and password file <none>
I20260927 03:28:02.710899 1860 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:02.710968 1860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:02.711100 1860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:02.712299 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-2/data/instance:
uuid: "6f9d13d5dae44a0da997d2540d7c7f1d"
format_stamp: "Formatted at 2026-09-27 03:28:02 on dist-test-slave-j0hf"
server_key: "0b5de80ed7bac001c21c701ae98338cc"
server_key_iv: "9e139df30aaea5fb0f80d7c6d33189d4"
server_key_version: "encryptionkey@0"
I20260927 03:28:02.712674 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6f9d13d5dae44a0da997d2540d7c7f1d"
format_stamp: "Formatted at 2026-09-27 03:28:02 on dist-test-slave-j0hf"
server_key: "0b5de80ed7bac001c21c701ae98338cc"
server_key_iv: "9e139df30aaea5fb0f80d7c6d33189d4"
server_key_version: "encryptionkey@0"
I20260927 03:28:02.714006 1860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:28:02.714717 1875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:02.714879 1860 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:02.714975 1860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "6f9d13d5dae44a0da997d2540d7c7f1d"
format_stamp: "Formatted at 2026-09-27 03:28:02 on dist-test-slave-j0hf"
server_key: "0b5de80ed7bac001c21c701ae98338cc"
server_key_iv: "9e139df30aaea5fb0f80d7c6d33189d4"
server_key_version: "encryptionkey@0"
I20260927 03:28:02.715056 1860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:02.725111 1860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:02.727499 1860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:02.727650 1860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:02.728238 1860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:02.728287 1860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:02.728327 1860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:02.728346 1860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 03:28:02 dist-test-slave-j0hf krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:02.735687 1860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46091
I20260927 03:28:02.735769 1990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46091 every 8 connection(s)
I20260927 03:28:02.736123 1860 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:28:02.738361 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:02.728621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53213 (local address 127.0.98.62:40781)
0927 03:28:02.728686 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:02.728689 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:02.728918 (+ 229us) server_negotiation.cc:407] Connection header received
0927 03:28:02.729481 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:02.729482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:02.729497 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:02.729584 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:02.730207 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.730974 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.732060 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.732269 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.732354 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:02.735315 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:02.735316 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:02.735316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:02.735323 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:02.736959 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:02.737384 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:02.737385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:02.737385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:02.737441 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:02.737739 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:02.737739 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:02.737740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:02.737853 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:02.737903 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:02.738218 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 03:28:02.738288 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 03:28:02.738598 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:02.728629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40781 (local address 127.0.98.3:53213)
0927 03:28:02.728787 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:02.728826 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:02.729388 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:02.729665 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:02.729670 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:02.729911 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:02.730094 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:02.730098 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.731067 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.731069 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:02.731955 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:02.731958 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.732363 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.732364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:02.732490 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:02.733003 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:02.733011 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:02.735225 (+ 2214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:02.737063 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:02.737069 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:02.737071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:02.737309 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:02.737538 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:02.737539 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:02.737539 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:02.737665 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:02.737936 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:02.737941 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:02.738077 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 03:28:02.738183 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 03:28:02.738324 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":93,"threads_started":1}
I20260927 03:28:02.739122 1991 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40781
I20260927 03:28:02.739219 1991 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:02.739502 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 1860
I20260927 03:28:02.739627 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:28:02.739840 392 external_mini_cluster.cc:1428] Setting key 2177c224fd90ea2be8365a30c3a912e6
I20260927 03:28:02.740192 1991 heartbeater.cc:511] Master 127.0.98.62:40781 requested a full tablet report, sending...
I20260927 03:28:02.740628 1507 ts_manager.cc:194] Registered new tserver with Master: 6f9d13d5dae44a0da997d2540d7c7f1d (127.0.98.3:46091)
I20260927 03:28:02.741247 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53213
I20260927 03:28:02.746273 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:02.740171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40781 (local address 127.0.98.3:60325)
0927 03:28:02.740345 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:02.740358 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:02.740414 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:02.740571 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:02.740572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:02.740760 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:02.740843 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:02.740845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.741879 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.741880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:02.742425 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:02.742427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.742777 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.742777 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:02.742825 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:02.743262 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:02.743266 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:02.743655 (+ 389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:02.745304 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:02.745305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:02.745305 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:02.745465 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:02.745708 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:02.745708 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:02.745709 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:02.745760 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:02.745978 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:02.745979 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:02.746034 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 03:28:02.746116 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 03:28:02.746198 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":75,"threads_started":1}
I20260927 03:28:02.746273 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:02.740220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60325 (local address 127.0.98.62:40781)
0927 03:28:02.740323 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:02.740325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:02.740404 (+ 79us) server_negotiation.cc:407] Connection header received
0927 03:28:02.740470 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:02.740471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:02.740484 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:02.740566 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:02.741007 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.741784 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.742522 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.742705 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.742770 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:02.743744 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:02.743746 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:02.743746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:02.743752 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:02.745227 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:02.745583 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:02.745584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:02.745584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:02.745641 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:02.745803 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:02.745803 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:02.745804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:02.745898 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:02.745945 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:02.746150 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 03:28:02.746211 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 03:28:02.751874 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 03:28:02 dist-test-slave-j0hf krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479682, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 03:28:02 dist-test-slave-j0hf krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479682, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:02.775348 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:02.767962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49952 (local address 127.0.98.62:40781)
0927 03:28:02.768040 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:02.768042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:02.768051 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:28:02.768084 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:02.768085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:02.768093 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:02.768164 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:02.768768 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.769278 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.770068 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.770258 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.770336 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:02.772705 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:02.772706 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:02.772707 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:02.772714 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:02.774086 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:02.774488 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:02.774490 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:02.774490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:02.774559 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:02.774762 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:02.774763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:02.774763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:02.774887 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:02.774988 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:02.775150 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 03:28:02.775236 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 03:28:02.777092 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:28:02.778685 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:28:02.785485 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:02.781075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37755 (local address 127.0.0.1:57814)
0927 03:28:02.781332 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:02.781342 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:02.781400 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:02.781874 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:02.781875 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:02.781876 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:02.781920 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:02.781921 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.783208 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.783208 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:02.784465 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:02.784467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.785157 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.785158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:02.785249 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:02.785265 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 03:28:02.785335 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 03:28:02.785389 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":101,"threads_started":1}
I20260927 03:28:02.785858 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:02.781520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57814 (local address 127.0.98.2:37755)
0927 03:28:02.781702 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:02.781707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:02.781716 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:28:02.781752 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:02.781754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:02.781769 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:02.782015 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:02.782068 (+ 53us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.783135 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.784547 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.785045 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.785154 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:02.785237 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:02.785713 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 03:28:02.785764 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":83,"threads_started":1}
I20260927 03:28:02.786410 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:02.781060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37057 (local address 127.0.0.1:44088)
0927 03:28:02.781164 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:02.781207 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:02.781272 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:02.781680 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:02.781684 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:02.781686 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:02.781787 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:02.781789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.784578 (+ 2789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.784579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:02.785253 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:02.785255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.786185 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.786186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:02.786255 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:02.786259 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:28:02.786289 (+ 30us) client_negotiation.cc:240] Negotiation successful
0927 03:28:02.786329 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":58,"threads_started":1}
I20260927 03:28:02.786412 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:02.781079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46091 (local address 127.0.0.1:45416)
0927 03:28:02.781460 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:02.781467 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:02.781497 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:02.782069 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:02.782070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:02.782072 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:02.782164 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:02.782165 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.783058 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.783059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:02.784464 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:02.784467 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.786173 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.786174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:02.786259 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:02.786264 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:28:02.786296 (+ 32us) client_negotiation.cc:240] Negotiation successful
0927 03:28:02.786347 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":29,"spinlock_wait_cycles":768,"thread_start_us":101,"threads_started":1}
I20260927 03:28:02.786580 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 7dc1e0cd32024e728865e8e243986283 (DEFAULT_TABLE table=test-table [id=e56ee8b0d6284cc291ea909b4c151160]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:02.786736 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:02.781692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45416 (local address 127.0.98.3:46091)
0927 03:28:02.781894 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:02.781899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:02.781908 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:28:02.781948 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:02.781950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:02.781965 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:02.782068 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:02.782246 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.782973 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.785274 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.786081 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.786487 (+ 406us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:02.786585 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:02.786615 (+ 30us) server_negotiation.cc:299] Negotiation successful
0927 03:28:02.786659 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":75,"threads_started":1}
I20260927 03:28:02.786772 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:02.781326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44088 (local address 127.0.98.1:37057)
0927 03:28:02.781483 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:02.781488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:02.781499 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:28:02.781540 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:02.781542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:02.781559 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:02.781645 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:02.781941 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.783124 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.785348 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.786086 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.786522 (+ 436us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:02.786612 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:02.786640 (+ 28us) server_negotiation.cc:299] Negotiation successful
0927 03:28:02.786681 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260927 03:28:02.786991 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dc1e0cd32024e728865e8e243986283. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:02.787369 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 7dc1e0cd32024e728865e8e243986283 (DEFAULT_TABLE table=test-table [id=e56ee8b0d6284cc291ea909b4c151160]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:02.787724 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dc1e0cd32024e728865e8e243986283. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:02.788190 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 7dc1e0cd32024e728865e8e243986283 (DEFAULT_TABLE table=test-table [id=e56ee8b0d6284cc291ea909b4c151160]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:02.788514 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dc1e0cd32024e728865e8e243986283. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:02.791069 2014 tablet_bootstrap.cc:492] T 7dc1e0cd32024e728865e8e243986283 P 5803b8bb7755400e8d5c81a565fa588a: Bootstrap starting.
I20260927 03:28:02.791342 2015 tablet_bootstrap.cc:492] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d: Bootstrap starting.
I20260927 03:28:02.791736 2016 tablet_bootstrap.cc:492] T 7dc1e0cd32024e728865e8e243986283 P 6fadb3b6c4c04523806c27d488b1cec2: Bootstrap starting.
I20260927 03:28:02.791822 2014 tablet_bootstrap.cc:654] T 7dc1e0cd32024e728865e8e243986283 P 5803b8bb7755400e8d5c81a565fa588a: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:02.792024 2015 tablet_bootstrap.cc:654] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:02.792156 2014 log.cc:824] T 7dc1e0cd32024e728865e8e243986283 P 5803b8bb7755400e8d5c81a565fa588a: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:02.792361 2015 log.cc:824] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:02.792411 2016 tablet_bootstrap.cc:654] T 7dc1e0cd32024e728865e8e243986283 P 6fadb3b6c4c04523806c27d488b1cec2: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:02.792692 2016 log.cc:824] T 7dc1e0cd32024e728865e8e243986283 P 6fadb3b6c4c04523806c27d488b1cec2: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:02.793021 2014 tablet_bootstrap.cc:492] T 7dc1e0cd32024e728865e8e243986283 P 5803b8bb7755400e8d5c81a565fa588a: No bootstrap required, opened a new log
I20260927 03:28:02.793077 2014 ts_tablet_manager.cc:1397] T 7dc1e0cd32024e728865e8e243986283 P 5803b8bb7755400e8d5c81a565fa588a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:28:02.793373 2016 tablet_bootstrap.cc:492] T 7dc1e0cd32024e728865e8e243986283 P 6fadb3b6c4c04523806c27d488b1cec2: No bootstrap required, opened a new log
I20260927 03:28:02.793426 2016 ts_tablet_manager.cc:1397] T 7dc1e0cd32024e728865e8e243986283 P 6fadb3b6c4c04523806c27d488b1cec2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:28:02.794981 2014 raft_consensus.cc:348] T 7dc1e0cd32024e728865e8e243986283 P 5803b8bb7755400e8d5c81a565fa588a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } }
I20260927 03:28:02.795141 2014 raft_consensus.cc:374] T 7dc1e0cd32024e728865e8e243986283 P 5803b8bb7755400e8d5c81a565fa588a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:02.795186 2014 raft_consensus.cc:729] T 7dc1e0cd32024e728865e8e243986283 P 5803b8bb7755400e8d5c81a565fa588a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5803b8bb7755400e8d5c81a565fa588a, State: Initialized, Role: FOLLOWER
I20260927 03:28:02.795307 2014 consensus_queue.cc:260] T 7dc1e0cd32024e728865e8e243986283 P 5803b8bb7755400e8d5c81a565fa588a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } }
I20260927 03:28:02.795248 2016 raft_consensus.cc:348] T 7dc1e0cd32024e728865e8e243986283 P 6fadb3b6c4c04523806c27d488b1cec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } }
I20260927 03:28:02.795387 2016 raft_consensus.cc:374] T 7dc1e0cd32024e728865e8e243986283 P 6fadb3b6c4c04523806c27d488b1cec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:02.795409 2016 raft_consensus.cc:729] T 7dc1e0cd32024e728865e8e243986283 P 6fadb3b6c4c04523806c27d488b1cec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fadb3b6c4c04523806c27d488b1cec2, State: Initialized, Role: FOLLOWER
I20260927 03:28:02.795574 2016 consensus_queue.cc:260] T 7dc1e0cd32024e728865e8e243986283 P 6fadb3b6c4c04523806c27d488b1cec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } }
I20260927 03:28:02.795629 2014 ts_tablet_manager.cc:1428] T 7dc1e0cd32024e728865e8e243986283 P 5803b8bb7755400e8d5c81a565fa588a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 03:28:02.795681 1851 heartbeater.cc:503] Master 127.0.98.62:40781 was elected leader, sending a full tablet report...
I20260927 03:28:02.795828 2016 ts_tablet_manager.cc:1428] T 7dc1e0cd32024e728865e8e243986283 P 6fadb3b6c4c04523806c27d488b1cec2: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 03:28:02.795938 1711 heartbeater.cc:503] Master 127.0.98.62:40781 was elected leader, sending a full tablet report...
I20260927 03:28:02.796088 2015 tablet_bootstrap.cc:492] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d: No bootstrap required, opened a new log
I20260927 03:28:02.796144 2015 ts_tablet_manager.cc:1397] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 03:28:02.797863 2015 raft_consensus.cc:348] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } }
I20260927 03:28:02.798023 2015 raft_consensus.cc:374] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:02.798058 2015 raft_consensus.cc:729] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f9d13d5dae44a0da997d2540d7c7f1d, State: Initialized, Role: FOLLOWER
I20260927 03:28:02.798166 2015 consensus_queue.cc:260] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } }
I20260927 03:28:02.798439 2015 ts_tablet_manager.cc:1428] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 03:28:02.798452 1991 heartbeater.cc:503] Master 127.0.98.62:40781 was elected leader, sending a full tablet report...
W20260927 03:28:02.828938 1852 tablet.cc:2369] T 7dc1e0cd32024e728865e8e243986283 P 5803b8bb7755400e8d5c81a565fa588a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:28:02.922163 1712 tablet.cc:2369] T 7dc1e0cd32024e728865e8e243986283 P 6fadb3b6c4c04523806c27d488b1cec2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:28:02.934055 2022 raft_consensus.cc:482] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:28:02.934188 2022 raft_consensus.cc:504] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } }
I20260927 03:28:02.934614 2022 leader_election.cc:288] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5803b8bb7755400e8d5c81a565fa588a (127.0.98.2:37755), 6fadb3b6c4c04523806c27d488b1cec2 (127.0.98.1:37057)
I20260927 03:28:02.935231 2021 raft_consensus.cc:482] T 7dc1e0cd32024e728865e8e243986283 P 6fadb3b6c4c04523806c27d488b1cec2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:28:02.935333 2021 raft_consensus.cc:504] T 7dc1e0cd32024e728865e8e243986283 P 6fadb3b6c4c04523806c27d488b1cec2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } }
I20260927 03:28:02.935714 2021 leader_election.cc:288] T 7dc1e0cd32024e728865e8e243986283 P 6fadb3b6c4c04523806c27d488b1cec2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5803b8bb7755400e8d5c81a565fa588a (127.0.98.2:37755), 6f9d13d5dae44a0da997d2540d7c7f1d (127.0.98.3:46091)
I20260927 03:28:02.938257 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:02.934931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53669 (local address 127.0.98.1:37057)
0927 03:28:02.934977 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:02.934979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:02.935139 (+ 160us) server_negotiation.cc:407] Connection header received
0927 03:28:02.935195 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:02.935196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:02.935207 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:02.935262 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:02.935409 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.936118 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.937575 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.937930 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.937964 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:02.937972 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:02.938159 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 03:28:02.938192 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:28:02.938275 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:02.934837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37057 (local address 127.0.98.3:53669)
0927 03:28:02.935043 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:02.935051 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:02.935136 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:02.935265 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:02.935265 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:02.935267 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:02.935336 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:02.935337 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.936202 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.936203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:02.937468 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:02.937471 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.938016 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.938017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:02.938071 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:02.938074 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:28:02.938153 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 03:28:02.938192 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":51,"thread_start_us":84,"threads_started":1}
I20260927 03:28:02.938431 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:02.934830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37755 (local address 127.0.98.3:56589)
0927 03:28:02.934884 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:02.934898 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:02.934950 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:02.935080 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:02.935081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:02.935082 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:02.935152 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:02.935155 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.936241 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.936242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:02.937468 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:02.937471 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.938182 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.938183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:02.938255 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:02.938258 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:28:02.938313 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 03:28:02.938347 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":69}
I20260927 03:28:02.938598 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc1e0cd32024e728865e8e243986283" candidate_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fadb3b6c4c04523806c27d488b1cec2" is_pre_election: true
I20260927 03:28:02.938719 1665 raft_consensus.cc:2454] T 7dc1e0cd32024e728865e8e243986283 P 6fadb3b6c4c04523806c27d488b1cec2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f9d13d5dae44a0da997d2540d7c7f1d in term 0.
I20260927 03:28:02.938824 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:02.934920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56589 (local address 127.0.98.2:37755)
0927 03:28:02.934965 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:02.934967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:02.934977 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:28:02.935006 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:02.935007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:02.935018 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:02.935075 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:02.935233 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.936025 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.937552 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.938076 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.938143 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:02.938152 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:02.938723 (+ 571us) server_negotiation.cc:299] Negotiation successful
0927 03:28:02.938763 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:28:02.939108 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc1e0cd32024e728865e8e243986283" candidate_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5803b8bb7755400e8d5c81a565fa588a" is_pre_election: true
I20260927 03:28:02.939234 1805 raft_consensus.cc:2454] T 7dc1e0cd32024e728865e8e243986283 P 5803b8bb7755400e8d5c81a565fa588a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f9d13d5dae44a0da997d2540d7c7f1d in term 0.
I20260927 03:28:02.939257 1876 leader_election.cc:302] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f9d13d5dae44a0da997d2540d7c7f1d, 6fadb3b6c4c04523806c27d488b1cec2; no voters:
I20260927 03:28:02.939395 2022 raft_consensus.cc:2781] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:28:02.939462 2022 raft_consensus.cc:482] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:28:02.939494 2022 raft_consensus.cc:3037] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:02.939682 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:02.935826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37755 (local address 127.0.98.1:49973)
0927 03:28:02.936148 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:02.936158 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:02.936210 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:02.936443 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:02.936443 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:02.936444 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:02.936502 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:02.936505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.937417 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.937418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:02.938582 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:02.938585 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.939455 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.939456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:02.939524 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:02.939528 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:28:02.939558 (+ 30us) client_negotiation.cc:240] Negotiation successful
0927 03:28:02.939596 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":2,"thread_start_us":49,"threads_started":1}
I20260927 03:28:02.939957 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:02.936009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49973 (local address 127.0.98.2:37755)
0927 03:28:02.936320 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:02.936321 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:02.936328 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:28:02.936347 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:02.936348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:02.936357 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:02.936566 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:02.936598 (+ 32us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.937334 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.938697 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.939346 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.939791 (+ 445us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:02.939807 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:02.939828 (+ 21us) server_negotiation.cc:299] Negotiation successful
0927 03:28:02.939861 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":78,"threads_started":1}
I20260927 03:28:02.940143 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc1e0cd32024e728865e8e243986283" candidate_uuid: "6fadb3b6c4c04523806c27d488b1cec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5803b8bb7755400e8d5c81a565fa588a" is_pre_election: true
I20260927 03:28:02.940234 1805 raft_consensus.cc:2454] T 7dc1e0cd32024e728865e8e243986283 P 5803b8bb7755400e8d5c81a565fa588a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fadb3b6c4c04523806c27d488b1cec2 in term 0.
I20260927 03:28:02.940407 1598 leader_election.cc:302] T 7dc1e0cd32024e728865e8e243986283 P 6fadb3b6c4c04523806c27d488b1cec2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5803b8bb7755400e8d5c81a565fa588a, 6fadb3b6c4c04523806c27d488b1cec2; no voters:
I20260927 03:28:02.940404 2022 raft_consensus.cc:504] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } }
I20260927 03:28:02.940541 2022 leader_election.cc:288] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d [CANDIDATE]: Term 1 election: Requested vote from peers 5803b8bb7755400e8d5c81a565fa588a (127.0.98.2:37755), 6fadb3b6c4c04523806c27d488b1cec2 (127.0.98.1:37057)
I20260927 03:28:02.940558 2021 raft_consensus.cc:2781] T 7dc1e0cd32024e728865e8e243986283 P 6fadb3b6c4c04523806c27d488b1cec2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:28:02.940613 2021 raft_consensus.cc:482] T 7dc1e0cd32024e728865e8e243986283 P 6fadb3b6c4c04523806c27d488b1cec2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:28:02.940646 2021 raft_consensus.cc:3037] T 7dc1e0cd32024e728865e8e243986283 P 6fadb3b6c4c04523806c27d488b1cec2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:02.940737 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc1e0cd32024e728865e8e243986283" candidate_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fadb3b6c4c04523806c27d488b1cec2"
I20260927 03:28:02.940973 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:02.936183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34143 (local address 127.0.98.3:46091)
0927 03:28:02.936219 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:02.936221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:02.936251 (+ 30us) server_negotiation.cc:407] Connection header received
0927 03:28:02.936347 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:02.936348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:02.936357 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:02.936388 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:02.936609 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.937314 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.940106 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.940626 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.940667 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:02.940675 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:02.940881 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 03:28:02.940917 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":186,"server-negotiator.queue_time_us":21}
I20260927 03:28:02.941203 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:02.936098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46091 (local address 127.0.98.1:34143)
0927 03:28:02.936197 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:02.936205 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:02.936272 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:02.936436 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:02.936437 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:02.936439 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:02.936513 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:02.936515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.937404 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.937405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:02.939996 (+ 2591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:02.939999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.940736 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.940736 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:02.940805 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:02.940809 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:28:02.941065 (+ 256us) client_negotiation.cc:240] Negotiation successful
0927 03:28:02.941118 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":1234,"thread_start_us":55,"threads_started":1}
I20260927 03:28:02.941489 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc1e0cd32024e728865e8e243986283" candidate_uuid: "6fadb3b6c4c04523806c27d488b1cec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" is_pre_election: true
I20260927 03:28:02.941519 2021 raft_consensus.cc:504] T 7dc1e0cd32024e728865e8e243986283 P 6fadb3b6c4c04523806c27d488b1cec2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } }
I20260927 03:28:02.941612 1945 raft_consensus.cc:2379] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6fadb3b6c4c04523806c27d488b1cec2 in current term 1: Already voted for candidate 6f9d13d5dae44a0da997d2540d7c7f1d in this term.
I20260927 03:28:02.941645 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc1e0cd32024e728865e8e243986283" candidate_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5803b8bb7755400e8d5c81a565fa588a"
I20260927 03:28:02.941690 2021 leader_election.cc:288] T 7dc1e0cd32024e728865e8e243986283 P 6fadb3b6c4c04523806c27d488b1cec2 [CANDIDATE]: Term 1 election: Requested vote from peers 5803b8bb7755400e8d5c81a565fa588a (127.0.98.2:37755), 6f9d13d5dae44a0da997d2540d7c7f1d (127.0.98.3:46091)
I20260927 03:28:02.941715 1805 raft_consensus.cc:3037] T 7dc1e0cd32024e728865e8e243986283 P 5803b8bb7755400e8d5c81a565fa588a [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:02.941862 1665 raft_consensus.cc:2379] T 7dc1e0cd32024e728865e8e243986283 P 6fadb3b6c4c04523806c27d488b1cec2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6f9d13d5dae44a0da997d2540d7c7f1d in current term 1: Already voted for candidate 6fadb3b6c4c04523806c27d488b1cec2 in this term.
I20260927 03:28:02.941960 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc1e0cd32024e728865e8e243986283" candidate_uuid: "6fadb3b6c4c04523806c27d488b1cec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d"
I20260927 03:28:02.942031 1945 raft_consensus.cc:2379] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6fadb3b6c4c04523806c27d488b1cec2 in current term 1: Already voted for candidate 6f9d13d5dae44a0da997d2540d7c7f1d in this term.
I20260927 03:28:02.942034 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc1e0cd32024e728865e8e243986283" candidate_uuid: "6fadb3b6c4c04523806c27d488b1cec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5803b8bb7755400e8d5c81a565fa588a"
I20260927 03:28:02.942270 1598 leader_election.cc:302] T 7dc1e0cd32024e728865e8e243986283 P 6fadb3b6c4c04523806c27d488b1cec2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fadb3b6c4c04523806c27d488b1cec2; no voters: 5803b8bb7755400e8d5c81a565fa588a, 6f9d13d5dae44a0da997d2540d7c7f1d
I20260927 03:28:02.942435 2021 raft_consensus.cc:2726] T 7dc1e0cd32024e728865e8e243986283 P 6fadb3b6c4c04523806c27d488b1cec2 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 03:28:02.942667 1805 raft_consensus.cc:2454] T 7dc1e0cd32024e728865e8e243986283 P 5803b8bb7755400e8d5c81a565fa588a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f9d13d5dae44a0da997d2540d7c7f1d in term 1.
I20260927 03:28:02.942821 1878 leader_election.cc:302] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5803b8bb7755400e8d5c81a565fa588a, 6f9d13d5dae44a0da997d2540d7c7f1d; no voters: 6fadb3b6c4c04523806c27d488b1cec2
I20260927 03:28:02.942921 2022 raft_consensus.cc:2781] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:02.943077 2022 raft_consensus.cc:686] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d [term 1 LEADER]: Becoming Leader. State: Replica: 6f9d13d5dae44a0da997d2540d7c7f1d, State: Running, Role: LEADER
I20260927 03:28:02.943167 2022 consensus_queue.cc:237] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } }
I20260927 03:28:02.944041 1508 catalog_manager.cc:5495] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f9d13d5dae44a0da997d2540d7c7f1d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } health_report { overall_health: HEALTHY } } }
I20260927 03:28:02.967211 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:02.964584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45418 (local address 127.0.98.3:46091)
0927 03:28:02.964648 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:02.964650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:02.964660 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:28:02.964771 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:02.964772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:02.964799 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:02.964874 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:28:02.965023 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.965505 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.966342 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:02.966538 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:02.966620 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:02.966848 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:28:02.966952 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:28:02.967038 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:02.967091 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 03:28:02.967115 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 03:28:02.969141 1665 raft_consensus.cc:1261] T 7dc1e0cd32024e728865e8e243986283 P 6fadb3b6c4c04523806c27d488b1cec2 [term 1 FOLLOWER]: Refusing update from remote peer 6f9d13d5dae44a0da997d2540d7c7f1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:02.969188 1805 raft_consensus.cc:1261] T 7dc1e0cd32024e728865e8e243986283 P 5803b8bb7755400e8d5c81a565fa588a [term 1 FOLLOWER]: Refusing update from remote peer 6f9d13d5dae44a0da997d2540d7c7f1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:02.969467 2022 consensus_queue.cc:1035] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:02.969698 2027 consensus_queue.cc:1035] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:02.972095 2030 mvcc.cc:205] Tried to move back new op lower bound from 7333804781439090688 to 7333804781335916544. Current Snapshot: MvccSnapshot[applied={T|T < 7333804781439090688}]
I20260927 03:28:02.972303 2031 mvcc.cc:205] Tried to move back new op lower bound from 7333804781439090688 to 7333804781335916544. Current Snapshot: MvccSnapshot[applied={T|T < 7333804781439090688}]
I20260927 03:28:02.973217 2032 mvcc.cc:205] Tried to move back new op lower bound from 7333804781439090688 to 7333804781335916544. Current Snapshot: MvccSnapshot[applied={T|T < 7333804781439090688 or (T in {7333804781439090688})}]
I20260927 03:28:02.976306 1508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:28:02.976397 1508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:28:02.977193 1508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc115db5035545bdb6cc4d97816cf042: Sending DeleteTablet for 3 replicas of tablet 7dc1e0cd32024e728865e8e243986283
I20260927 03:28:02.977491 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dc1e0cd32024e728865e8e243986283 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44088
I20260927 03:28:02.977494 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dc1e0cd32024e728865e8e243986283 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57814
I20260927 03:28:02.977705 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dc1e0cd32024e728865e8e243986283 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45416
I20260927 03:28:02.977836 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 03:28:02.977923 2041 tablet_replica.cc:330] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d: stopping tablet replica
W20260927 03:28:02.977973 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:28:02.978015 2041 raft_consensus.cc:2229] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:28:02.978106 2041 raft_consensus.cc:2258] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:02.978395 2040 tablet_replica.cc:330] T 7dc1e0cd32024e728865e8e243986283 P 6fadb3b6c4c04523806c27d488b1cec2: stopping tablet replica
I20260927 03:28:02.978505 2040 raft_consensus.cc:2229] T 7dc1e0cd32024e728865e8e243986283 P 6fadb3b6c4c04523806c27d488b1cec2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:28:02.978658 2040 raft_consensus.cc:2258] T 7dc1e0cd32024e728865e8e243986283 P 6fadb3b6c4c04523806c27d488b1cec2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:02.978693 2041 ts_tablet_manager.cc:1905] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:02.979341 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 00f61fbe663e4356830533b4871ceede (DEFAULT_TABLE table=test-table [id=0bb849292c81455b8f5b2b0b07c11388]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:02.979498 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00f61fbe663e4356830533b4871ceede. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:02.980573 2041 ts_tablet_manager.cc:1918] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:28:02.980655 2041 log.cc:1196] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-2/wal/wals/7dc1e0cd32024e728865e8e243986283
I20260927 03:28:02.980916 2041 ts_tablet_manager.cc:1939] T 7dc1e0cd32024e728865e8e243986283 P 6f9d13d5dae44a0da997d2540d7c7f1d: Deleting consensus metadata
I20260927 03:28:02.980996 2016 tablet_bootstrap.cc:492] T 00f61fbe663e4356830533b4871ceede P 6fadb3b6c4c04523806c27d488b1cec2: Bootstrap starting.
I20260927 03:28:02.981194 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 00f61fbe663e4356830533b4871ceede (DEFAULT_TABLE table=test-table [id=0bb849292c81455b8f5b2b0b07c11388]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:02.981310 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00f61fbe663e4356830533b4871ceede. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:02.981715 2016 tablet_bootstrap.cc:654] T 00f61fbe663e4356830533b4871ceede P 6fadb3b6c4c04523806c27d488b1cec2: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:02.982285 2014 tablet_bootstrap.cc:492] T 00f61fbe663e4356830533b4871ceede P 5803b8bb7755400e8d5c81a565fa588a: Bootstrap starting.
I20260927 03:28:02.982383 2016 tablet_bootstrap.cc:492] T 00f61fbe663e4356830533b4871ceede P 6fadb3b6c4c04523806c27d488b1cec2: No bootstrap required, opened a new log
I20260927 03:28:02.982426 2016 ts_tablet_manager.cc:1397] T 00f61fbe663e4356830533b4871ceede P 6fadb3b6c4c04523806c27d488b1cec2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 03:28:02.982512 1493 catalog_manager.cc:4841] TS 6f9d13d5dae44a0da997d2540d7c7f1d (127.0.98.3:46091): tablet 7dc1e0cd32024e728865e8e243986283 (table test-table [id=e56ee8b0d6284cc291ea909b4c151160]) successfully deleted
I20260927 03:28:02.982569 2016 raft_consensus.cc:348] T 00f61fbe663e4356830533b4871ceede P 6fadb3b6c4c04523806c27d488b1cec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } }
I20260927 03:28:02.982631 2016 raft_consensus.cc:374] T 00f61fbe663e4356830533b4871ceede P 6fadb3b6c4c04523806c27d488b1cec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:02.982652 2016 raft_consensus.cc:729] T 00f61fbe663e4356830533b4871ceede P 6fadb3b6c4c04523806c27d488b1cec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fadb3b6c4c04523806c27d488b1cec2, State: Initialized, Role: FOLLOWER
I20260927 03:28:02.982703 2016 consensus_queue.cc:260] T 00f61fbe663e4356830533b4871ceede P 6fadb3b6c4c04523806c27d488b1cec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } }
I20260927 03:28:02.982733 2014 tablet_bootstrap.cc:654] T 00f61fbe663e4356830533b4871ceede P 5803b8bb7755400e8d5c81a565fa588a: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:02.982761 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 00f61fbe663e4356830533b4871ceede (DEFAULT_TABLE table=test-table [id=0bb849292c81455b8f5b2b0b07c11388]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:02.982803 2016 ts_tablet_manager.cc:1428] T 00f61fbe663e4356830533b4871ceede P 6fadb3b6c4c04523806c27d488b1cec2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:02.982857 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00f61fbe663e4356830533b4871ceede. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:02.983733 2039 tablet_replica.cc:330] T 7dc1e0cd32024e728865e8e243986283 P 5803b8bb7755400e8d5c81a565fa588a: stopping tablet replica
I20260927 03:28:02.983839 2039 raft_consensus.cc:2229] T 7dc1e0cd32024e728865e8e243986283 P 5803b8bb7755400e8d5c81a565fa588a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:28:02.983959 2039 raft_consensus.cc:2258] T 7dc1e0cd32024e728865e8e243986283 P 5803b8bb7755400e8d5c81a565fa588a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:02.984081 2015 tablet_bootstrap.cc:492] T 00f61fbe663e4356830533b4871ceede P 6f9d13d5dae44a0da997d2540d7c7f1d: Bootstrap starting.
I20260927 03:28:02.984169 2040 ts_tablet_manager.cc:1905] T 7dc1e0cd32024e728865e8e243986283 P 6fadb3b6c4c04523806c27d488b1cec2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:02.984460 2015 tablet_bootstrap.cc:654] T 00f61fbe663e4356830533b4871ceede P 6f9d13d5dae44a0da997d2540d7c7f1d: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:02.985677 2015 tablet_bootstrap.cc:492] T 00f61fbe663e4356830533b4871ceede P 6f9d13d5dae44a0da997d2540d7c7f1d: No bootstrap required, opened a new log
I20260927 03:28:02.985738 2015 ts_tablet_manager.cc:1397] T 00f61fbe663e4356830533b4871ceede P 6f9d13d5dae44a0da997d2540d7c7f1d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:28:02.985781 2040 ts_tablet_manager.cc:1918] T 7dc1e0cd32024e728865e8e243986283 P 6fadb3b6c4c04523806c27d488b1cec2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:28:02.985859 2040 log.cc:1196] T 7dc1e0cd32024e728865e8e243986283 P 6fadb3b6c4c04523806c27d488b1cec2: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-0/wal/wals/7dc1e0cd32024e728865e8e243986283
I20260927 03:28:02.985877 2015 raft_consensus.cc:348] T 00f61fbe663e4356830533b4871ceede P 6f9d13d5dae44a0da997d2540d7c7f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } }
I20260927 03:28:02.985950 2015 raft_consensus.cc:374] T 00f61fbe663e4356830533b4871ceede P 6f9d13d5dae44a0da997d2540d7c7f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:02.985976 2015 raft_consensus.cc:729] T 00f61fbe663e4356830533b4871ceede P 6f9d13d5dae44a0da997d2540d7c7f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f9d13d5dae44a0da997d2540d7c7f1d, State: Initialized, Role: FOLLOWER
I20260927 03:28:02.986034 2015 consensus_queue.cc:260] T 00f61fbe663e4356830533b4871ceede P 6f9d13d5dae44a0da997d2540d7c7f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } }
I20260927 03:28:02.986120 2015 ts_tablet_manager.cc:1428] T 00f61fbe663e4356830533b4871ceede P 6f9d13d5dae44a0da997d2540d7c7f1d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:02.986129 2040 ts_tablet_manager.cc:1939] T 7dc1e0cd32024e728865e8e243986283 P 6fadb3b6c4c04523806c27d488b1cec2: Deleting consensus metadata
I20260927 03:28:02.986600 1492 catalog_manager.cc:4841] TS 6fadb3b6c4c04523806c27d488b1cec2 (127.0.98.1:37057): tablet 7dc1e0cd32024e728865e8e243986283 (table test-table [id=e56ee8b0d6284cc291ea909b4c151160]) successfully deleted
I20260927 03:28:02.986667 2039 ts_tablet_manager.cc:1905] T 7dc1e0cd32024e728865e8e243986283 P 5803b8bb7755400e8d5c81a565fa588a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 03:28:02.987267 1992 tablet.cc:2369] T 00f61fbe663e4356830533b4871ceede P 6f9d13d5dae44a0da997d2540d7c7f1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:28:02.987802 2014 tablet_bootstrap.cc:492] T 00f61fbe663e4356830533b4871ceede P 5803b8bb7755400e8d5c81a565fa588a: No bootstrap required, opened a new log
I20260927 03:28:02.987854 2014 ts_tablet_manager.cc:1397] T 00f61fbe663e4356830533b4871ceede P 5803b8bb7755400e8d5c81a565fa588a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260927 03:28:02.988018 2014 raft_consensus.cc:348] T 00f61fbe663e4356830533b4871ceede P 5803b8bb7755400e8d5c81a565fa588a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } }
I20260927 03:28:02.988112 2014 raft_consensus.cc:374] T 00f61fbe663e4356830533b4871ceede P 5803b8bb7755400e8d5c81a565fa588a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:02.988124 2039 ts_tablet_manager.cc:1918] T 7dc1e0cd32024e728865e8e243986283 P 5803b8bb7755400e8d5c81a565fa588a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:28:02.988137 2014 raft_consensus.cc:729] T 00f61fbe663e4356830533b4871ceede P 5803b8bb7755400e8d5c81a565fa588a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5803b8bb7755400e8d5c81a565fa588a, State: Initialized, Role: FOLLOWER
I20260927 03:28:02.988198 2039 log.cc:1196] T 7dc1e0cd32024e728865e8e243986283 P 5803b8bb7755400e8d5c81a565fa588a: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-1/wal/wals/7dc1e0cd32024e728865e8e243986283
I20260927 03:28:02.988210 2014 consensus_queue.cc:260] T 00f61fbe663e4356830533b4871ceede P 5803b8bb7755400e8d5c81a565fa588a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } }
I20260927 03:28:02.988317 2014 ts_tablet_manager.cc:1428] T 00f61fbe663e4356830533b4871ceede P 5803b8bb7755400e8d5c81a565fa588a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 03:28:02.988462 2039 ts_tablet_manager.cc:1939] T 7dc1e0cd32024e728865e8e243986283 P 5803b8bb7755400e8d5c81a565fa588a: Deleting consensus metadata
I20260927 03:28:02.988847 1494 catalog_manager.cc:4841] TS 5803b8bb7755400e8d5c81a565fa588a (127.0.98.2:37755): tablet 7dc1e0cd32024e728865e8e243986283 (table test-table [id=e56ee8b0d6284cc291ea909b4c151160]) successfully deleted
I20260927 03:28:03.269238 2020 raft_consensus.cc:482] T 00f61fbe663e4356830533b4871ceede P 5803b8bb7755400e8d5c81a565fa588a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:28:03.269331 2020 raft_consensus.cc:504] T 00f61fbe663e4356830533b4871ceede P 5803b8bb7755400e8d5c81a565fa588a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } }
I20260927 03:28:03.269728 2020 leader_election.cc:288] T 00f61fbe663e4356830533b4871ceede P 5803b8bb7755400e8d5c81a565fa588a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fadb3b6c4c04523806c27d488b1cec2 (127.0.98.1:37057), 6f9d13d5dae44a0da997d2540d7c7f1d (127.0.98.3:46091)
I20260927 03:28:03.272691 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:03.270048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58341 (local address 127.0.98.3:46091)
0927 03:28:03.270098 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:03.270100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:03.270111 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:28:03.270180 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:03.270181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:03.270193 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:03.270243 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:03.270422 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:03.270901 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:03.271951 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:03.272320 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:03.272406 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:03.272417 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:03.272579 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 03:28:03.272616 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 03:28:03.272691 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:03.270001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38045 (local address 127.0.98.1:37057)
0927 03:28:03.270051 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:03.270053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:03.270151 (+ 98us) server_negotiation.cc:407] Connection header received
0927 03:28:03.270217 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:03.270218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:03.270227 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:03.270276 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:03.270486 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:03.270942 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:03.271965 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:03.272327 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:03.272414 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:03.272424 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:03.272574 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 03:28:03.272612 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:28:03.272713 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:03.269928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37057 (local address 127.0.98.2:38045)
0927 03:28:03.270060 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:03.270070 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:03.270141 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:03.270289 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:03.270290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:03.270292 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:03.270355 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:03.270357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:03.271057 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:03.271058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:03.271862 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:03.271865 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:03.272416 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:03.272417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:03.272495 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:03.272499 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:28:03.272567 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 03:28:03.272611 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":62,"threads_started":1}
I20260927 03:28:03.272734 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:03.269900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46091 (local address 127.0.98.2:58341)
0927 03:28:03.270037 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:03.270049 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:03.270113 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:03.270245 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:03.270246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:03.270247 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:03.270346 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:03.270348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:03.270996 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:03.270998 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:03.271856 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:03.271859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:03.272406 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:03.272407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:03.272491 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:03.272495 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:28:03.272570 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 03:28:03.272616 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":6,"thread_start_us":61,"threads_started":1}
I20260927 03:28:03.273221 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f61fbe663e4356830533b4871ceede" candidate_uuid: "5803b8bb7755400e8d5c81a565fa588a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fadb3b6c4c04523806c27d488b1cec2" is_pre_election: true
I20260927 03:28:03.273301 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f61fbe663e4356830533b4871ceede" candidate_uuid: "5803b8bb7755400e8d5c81a565fa588a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" is_pre_election: true
I20260927 03:28:03.273351 1665 raft_consensus.cc:2454] T 00f61fbe663e4356830533b4871ceede P 6fadb3b6c4c04523806c27d488b1cec2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5803b8bb7755400e8d5c81a565fa588a in term 0.
I20260927 03:28:03.273389 1945 raft_consensus.cc:2454] T 00f61fbe663e4356830533b4871ceede P 6f9d13d5dae44a0da997d2540d7c7f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5803b8bb7755400e8d5c81a565fa588a in term 0.
I20260927 03:28:03.273581 1736 leader_election.cc:302] T 00f61fbe663e4356830533b4871ceede P 5803b8bb7755400e8d5c81a565fa588a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5803b8bb7755400e8d5c81a565fa588a, 6fadb3b6c4c04523806c27d488b1cec2; no voters:
I20260927 03:28:03.273754 2020 raft_consensus.cc:2781] T 00f61fbe663e4356830533b4871ceede P 5803b8bb7755400e8d5c81a565fa588a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:28:03.273809 2020 raft_consensus.cc:482] T 00f61fbe663e4356830533b4871ceede P 5803b8bb7755400e8d5c81a565fa588a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:28:03.273825 2020 raft_consensus.cc:3037] T 00f61fbe663e4356830533b4871ceede P 5803b8bb7755400e8d5c81a565fa588a [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:03.274549 2020 raft_consensus.cc:504] T 00f61fbe663e4356830533b4871ceede P 5803b8bb7755400e8d5c81a565fa588a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } }
I20260927 03:28:03.274696 2020 leader_election.cc:288] T 00f61fbe663e4356830533b4871ceede P 5803b8bb7755400e8d5c81a565fa588a [CANDIDATE]: Term 1 election: Requested vote from peers 6fadb3b6c4c04523806c27d488b1cec2 (127.0.98.1:37057), 6f9d13d5dae44a0da997d2540d7c7f1d (127.0.98.3:46091)
I20260927 03:28:03.274842 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f61fbe663e4356830533b4871ceede" candidate_uuid: "5803b8bb7755400e8d5c81a565fa588a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fadb3b6c4c04523806c27d488b1cec2"
I20260927 03:28:03.274936 1665 raft_consensus.cc:3037] T 00f61fbe663e4356830533b4871ceede P 6fadb3b6c4c04523806c27d488b1cec2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:03.274889 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f61fbe663e4356830533b4871ceede" candidate_uuid: "5803b8bb7755400e8d5c81a565fa588a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d"
I20260927 03:28:03.274976 1945 raft_consensus.cc:3037] T 00f61fbe663e4356830533b4871ceede P 6f9d13d5dae44a0da997d2540d7c7f1d [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:03.275805 1665 raft_consensus.cc:2454] T 00f61fbe663e4356830533b4871ceede P 6fadb3b6c4c04523806c27d488b1cec2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5803b8bb7755400e8d5c81a565fa588a in term 1.
I20260927 03:28:03.275831 1945 raft_consensus.cc:2454] T 00f61fbe663e4356830533b4871ceede P 6f9d13d5dae44a0da997d2540d7c7f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5803b8bb7755400e8d5c81a565fa588a in term 1.
I20260927 03:28:03.275988 1736 leader_election.cc:302] T 00f61fbe663e4356830533b4871ceede P 5803b8bb7755400e8d5c81a565fa588a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5803b8bb7755400e8d5c81a565fa588a, 6fadb3b6c4c04523806c27d488b1cec2; no voters:
I20260927 03:28:03.276093 2020 raft_consensus.cc:2781] T 00f61fbe663e4356830533b4871ceede P 5803b8bb7755400e8d5c81a565fa588a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:03.276216 2020 raft_consensus.cc:686] T 00f61fbe663e4356830533b4871ceede P 5803b8bb7755400e8d5c81a565fa588a [term 1 LEADER]: Becoming Leader. State: Replica: 5803b8bb7755400e8d5c81a565fa588a, State: Running, Role: LEADER
I20260927 03:28:03.276290 2020 consensus_queue.cc:237] T 00f61fbe663e4356830533b4871ceede P 5803b8bb7755400e8d5c81a565fa588a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } }
I20260927 03:28:03.277030 1507 catalog_manager.cc:5495] T 00f61fbe663e4356830533b4871ceede P 5803b8bb7755400e8d5c81a565fa588a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5803b8bb7755400e8d5c81a565fa588a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "5803b8bb7755400e8d5c81a565fa588a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } health_report { overall_health: UNKNOWN } } }
Sep 27 03:28:03 dist-test-slave-j0hf krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:03.363770 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:03.356317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40781 (local address 127.0.0.1:49958)
0927 03:28:03.356667 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:03.356681 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:03.356755 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:03.357019 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:03.357020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:03.357347 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:03.357409 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:03.357411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:03.358230 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:03.358232 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:03.358770 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:03.358772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:03.359122 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:03.359123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:03.359171 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:03.359834 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:03.359838 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:03.361298 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:03.362767 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:03.362774 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:03.362776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:03.362981 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:03.363158 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:03.363158 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:03.363159 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:03.363214 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:03.363417 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:03.363475 (+ 58us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:03.363528 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 03:28:03.363585 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 03:28:03.363676 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":111,"threads_started":1}
I20260927 03:28:03.363826 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:03.356363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49958 (local address 127.0.98.62:40781)
0927 03:28:03.356862 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:03.356864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:03.356871 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:28:03.356892 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:03.356892 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:03.356902 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:03.357003 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:03.357516 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:03.358118 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:03.358845 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:03.359033 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:03.359088 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:03.361383 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:03.361385 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:03.361385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:03.361391 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:03.362690 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:03.363047 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:03.363047 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:03.363048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:03.363099 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:03.363265 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:03.363266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:03.363266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:03.363363 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:03.363411 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:03.363620 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 03:28:03.363725 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":454,"thread_start_us":71,"threads_started":1}
I20260927 03:28:03.365662 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49958:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 03:28:03.365844 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:28:03.367254 1925 tablet_service.cc:1467] Processing CreateTablet for tablet f70970525c094cecbc8b9d0c341c1f24 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f2efd27fc8d1454295b2e50a46e00135]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:28:03.367264 1785 tablet_service.cc:1467] Processing CreateTablet for tablet f70970525c094cecbc8b9d0c341c1f24 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f2efd27fc8d1454295b2e50a46e00135]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:28:03.367255 1645 tablet_service.cc:1467] Processing CreateTablet for tablet f70970525c094cecbc8b9d0c341c1f24 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f2efd27fc8d1454295b2e50a46e00135]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:28:03.367439 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f70970525c094cecbc8b9d0c341c1f24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:03.367439 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f70970525c094cecbc8b9d0c341c1f24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:03.367424 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f70970525c094cecbc8b9d0c341c1f24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:03.368633 2015 tablet_bootstrap.cc:492] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d: Bootstrap starting.
I20260927 03:28:03.368937 2014 tablet_bootstrap.cc:492] T f70970525c094cecbc8b9d0c341c1f24 P 5803b8bb7755400e8d5c81a565fa588a: Bootstrap starting.
I20260927 03:28:03.368942 2016 tablet_bootstrap.cc:492] T f70970525c094cecbc8b9d0c341c1f24 P 6fadb3b6c4c04523806c27d488b1cec2: Bootstrap starting.
I20260927 03:28:03.369019 2015 tablet_bootstrap.cc:654] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:03.369275 2014 tablet_bootstrap.cc:654] T f70970525c094cecbc8b9d0c341c1f24 P 5803b8bb7755400e8d5c81a565fa588a: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:03.369782 2016 tablet_bootstrap.cc:654] T f70970525c094cecbc8b9d0c341c1f24 P 6fadb3b6c4c04523806c27d488b1cec2: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:03.369889 2014 tablet_bootstrap.cc:492] T f70970525c094cecbc8b9d0c341c1f24 P 5803b8bb7755400e8d5c81a565fa588a: No bootstrap required, opened a new log
I20260927 03:28:03.369947 2014 ts_tablet_manager.cc:1397] T f70970525c094cecbc8b9d0c341c1f24 P 5803b8bb7755400e8d5c81a565fa588a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 03:28:03.370131 2014 raft_consensus.cc:348] T f70970525c094cecbc8b9d0c341c1f24 P 5803b8bb7755400e8d5c81a565fa588a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } }
I20260927 03:28:03.370229 2014 raft_consensus.cc:374] T f70970525c094cecbc8b9d0c341c1f24 P 5803b8bb7755400e8d5c81a565fa588a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:03.370263 2014 raft_consensus.cc:729] T f70970525c094cecbc8b9d0c341c1f24 P 5803b8bb7755400e8d5c81a565fa588a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5803b8bb7755400e8d5c81a565fa588a, State: Initialized, Role: FOLLOWER
I20260927 03:28:03.370311 2014 consensus_queue.cc:260] T f70970525c094cecbc8b9d0c341c1f24 P 5803b8bb7755400e8d5c81a565fa588a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } }
I20260927 03:28:03.370383 2016 tablet_bootstrap.cc:492] T f70970525c094cecbc8b9d0c341c1f24 P 6fadb3b6c4c04523806c27d488b1cec2: No bootstrap required, opened a new log
I20260927 03:28:03.370409 2014 ts_tablet_manager.cc:1428] T f70970525c094cecbc8b9d0c341c1f24 P 5803b8bb7755400e8d5c81a565fa588a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 03:28:03.370440 2016 ts_tablet_manager.cc:1397] T f70970525c094cecbc8b9d0c341c1f24 P 6fadb3b6c4c04523806c27d488b1cec2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:28:03.370498 2020 tablet_replica.cc:439] T f70970525c094cecbc8b9d0c341c1f24 P 5803b8bb7755400e8d5c81a565fa588a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } } }
I20260927 03:28:03.370594 2020 tablet_replica.cc:442] T f70970525c094cecbc8b9d0c341c1f24 P 5803b8bb7755400e8d5c81a565fa588a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:28:03.370594 2016 raft_consensus.cc:348] T f70970525c094cecbc8b9d0c341c1f24 P 6fadb3b6c4c04523806c27d488b1cec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } }
I20260927 03:28:03.370678 2016 raft_consensus.cc:374] T f70970525c094cecbc8b9d0c341c1f24 P 6fadb3b6c4c04523806c27d488b1cec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:03.370707 2016 raft_consensus.cc:729] T f70970525c094cecbc8b9d0c341c1f24 P 6fadb3b6c4c04523806c27d488b1cec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fadb3b6c4c04523806c27d488b1cec2, State: Initialized, Role: FOLLOWER
I20260927 03:28:03.370756 2016 consensus_queue.cc:260] T f70970525c094cecbc8b9d0c341c1f24 P 6fadb3b6c4c04523806c27d488b1cec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } }
I20260927 03:28:03.370841 2016 ts_tablet_manager.cc:1428] T f70970525c094cecbc8b9d0c341c1f24 P 6fadb3b6c4c04523806c27d488b1cec2: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 03:28:03.370970 2021 tablet_replica.cc:439] T f70970525c094cecbc8b9d0c341c1f24 P 6fadb3b6c4c04523806c27d488b1cec2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } } }
I20260927 03:28:03.371053 2021 tablet_replica.cc:442] T f70970525c094cecbc8b9d0c341c1f24 P 6fadb3b6c4c04523806c27d488b1cec2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:28:03.371419 2015 tablet_bootstrap.cc:492] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d: No bootstrap required, opened a new log
I20260927 03:28:03.371503 2015 ts_tablet_manager.cc:1397] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 03:28:03.371616 2015 raft_consensus.cc:348] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } }
I20260927 03:28:03.371697 2015 raft_consensus.cc:374] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:03.371728 2015 raft_consensus.cc:729] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f9d13d5dae44a0da997d2540d7c7f1d, State: Initialized, Role: FOLLOWER
I20260927 03:28:03.371785 2015 consensus_queue.cc:260] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } }
I20260927 03:28:03.371893 2015 ts_tablet_manager.cc:1428] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:03.371917 2027 tablet_replica.cc:439] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } } }
I20260927 03:28:03.372011 2027 tablet_replica.cc:442] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:28:03.714756 2027 raft_consensus.cc:482] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:28:03.714890 2027 raft_consensus.cc:504] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } }
I20260927 03:28:03.715112 2027 leader_election.cc:288] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fadb3b6c4c04523806c27d488b1cec2 (127.0.98.1:37057), 5803b8bb7755400e8d5c81a565fa588a (127.0.98.2:37755)
I20260927 03:28:03.715343 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f70970525c094cecbc8b9d0c341c1f24" candidate_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fadb3b6c4c04523806c27d488b1cec2" is_pre_election: true
I20260927 03:28:03.715499 1665 raft_consensus.cc:2454] T f70970525c094cecbc8b9d0c341c1f24 P 6fadb3b6c4c04523806c27d488b1cec2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f9d13d5dae44a0da997d2540d7c7f1d in term 0.
I20260927 03:28:03.715518 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f70970525c094cecbc8b9d0c341c1f24" candidate_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5803b8bb7755400e8d5c81a565fa588a" is_pre_election: true
I20260927 03:28:03.715664 1805 raft_consensus.cc:2454] T f70970525c094cecbc8b9d0c341c1f24 P 5803b8bb7755400e8d5c81a565fa588a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f9d13d5dae44a0da997d2540d7c7f1d in term 0.
I20260927 03:28:03.715677 1876 leader_election.cc:302] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f9d13d5dae44a0da997d2540d7c7f1d, 6fadb3b6c4c04523806c27d488b1cec2; no voters:
I20260927 03:28:03.715814 2027 raft_consensus.cc:2781] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:28:03.715854 2027 raft_consensus.cc:482] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:28:03.715875 2027 raft_consensus.cc:3037] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:03.734570 2020 consensus_queue.cc:1035] T 00f61fbe663e4356830533b4871ceede P 5803b8bb7755400e8d5c81a565fa588a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:03.736527 2047 consensus_queue.cc:1035] T 00f61fbe663e4356830533b4871ceede P 5803b8bb7755400e8d5c81a565fa588a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:03.747995 2027 raft_consensus.cc:504] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } }
I20260927 03:28:03.748226 2027 leader_election.cc:288] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d [CANDIDATE]: Term 1 election: Requested vote from peers 6fadb3b6c4c04523806c27d488b1cec2 (127.0.98.1:37057), 5803b8bb7755400e8d5c81a565fa588a (127.0.98.2:37755)
I20260927 03:28:03.748477 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f70970525c094cecbc8b9d0c341c1f24" candidate_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5803b8bb7755400e8d5c81a565fa588a"
I20260927 03:28:03.748517 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f70970525c094cecbc8b9d0c341c1f24" candidate_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fadb3b6c4c04523806c27d488b1cec2"
I20260927 03:28:03.748587 1805 raft_consensus.cc:3037] T f70970525c094cecbc8b9d0c341c1f24 P 5803b8bb7755400e8d5c81a565fa588a [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:03.748633 1665 raft_consensus.cc:3037] T f70970525c094cecbc8b9d0c341c1f24 P 6fadb3b6c4c04523806c27d488b1cec2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:03.749307 1805 raft_consensus.cc:2454] T f70970525c094cecbc8b9d0c341c1f24 P 5803b8bb7755400e8d5c81a565fa588a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f9d13d5dae44a0da997d2540d7c7f1d in term 1.
I20260927 03:28:03.749540 1878 leader_election.cc:302] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5803b8bb7755400e8d5c81a565fa588a, 6f9d13d5dae44a0da997d2540d7c7f1d; no voters:
I20260927 03:28:03.749660 2027 raft_consensus.cc:2781] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:03.749713 2027 raft_consensus.cc:686] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d [term 1 LEADER]: Becoming Leader. State: Replica: 6f9d13d5dae44a0da997d2540d7c7f1d, State: Running, Role: LEADER
I20260927 03:28:03.749586 1665 raft_consensus.cc:2454] T f70970525c094cecbc8b9d0c341c1f24 P 6fadb3b6c4c04523806c27d488b1cec2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f9d13d5dae44a0da997d2540d7c7f1d in term 1.
I20260927 03:28:03.749779 2027 consensus_queue.cc:237] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } }
I20260927 03:28:03.750098 2062 tablet_replica.cc:439] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d: TxnStatusTablet state changed. Reason: New leader 6f9d13d5dae44a0da997d2540d7c7f1d. Latest consensus state: current_term: 1 leader_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } } }
I20260927 03:28:03.750211 2062 tablet_replica.cc:442] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:28:03.750404 2066 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 03:28:03.750524 2066 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 03:28:03.750815 1507 catalog_manager.cc:5495] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f9d13d5dae44a0da997d2540d7c7f1d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } health_report { overall_health: HEALTHY } } }
I20260927 03:28:03.813774 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:03.811347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45426 (local address 127.0.98.3:46091)
0927 03:28:03.811472 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:03.811474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:03.811482 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:28:03.811508 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:03.811508 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:03.811525 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:03.811598 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:28:03.811746 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:03.812241 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:03.813016 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:03.813211 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:03.813287 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:03.813460 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:28:03.813499 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:28:03.813559 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:03.813626 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 03:28:03.813647 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":52,"threads_started":1}
I20260927 03:28:03.813751 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:03.811243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46091 (local address 127.0.0.1:45426)
0927 03:28:03.811327 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:03.811340 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:03.811396 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:03.811605 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:03.811606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:03.811607 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:28:03.811661 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:03.811664 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:03.812330 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:03.812331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:03.812919 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:03.812921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:03.813286 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:03.813287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:03.813344 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:03.813360 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:28:03.813562 (+ 202us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:28:03.813571 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 03:28:03.813614 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 03:28:03.813665 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 03:28:03.814968 1805 raft_consensus.cc:1261] T f70970525c094cecbc8b9d0c341c1f24 P 5803b8bb7755400e8d5c81a565fa588a [term 1 FOLLOWER]: Refusing update from remote peer 6f9d13d5dae44a0da997d2540d7c7f1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:03.815034 1665 raft_consensus.cc:1261] T f70970525c094cecbc8b9d0c341c1f24 P 6fadb3b6c4c04523806c27d488b1cec2 [term 1 FOLLOWER]: Refusing update from remote peer 6f9d13d5dae44a0da997d2540d7c7f1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:03.815210 2062 consensus_queue.cc:1035] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:03.815320 2027 consensus_queue.cc:1035] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:03.815804 2020 tablet_replica.cc:439] T f70970525c094cecbc8b9d0c341c1f24 P 5803b8bb7755400e8d5c81a565fa588a: TxnStatusTablet state changed. Reason: New leader 6f9d13d5dae44a0da997d2540d7c7f1d. Latest consensus state: current_term: 1 leader_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } } }
I20260927 03:28:03.815894 2020 tablet_replica.cc:442] T f70970525c094cecbc8b9d0c341c1f24 P 5803b8bb7755400e8d5c81a565fa588a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:28:03.815990 2021 tablet_replica.cc:439] T f70970525c094cecbc8b9d0c341c1f24 P 6fadb3b6c4c04523806c27d488b1cec2: TxnStatusTablet state changed. Reason: New leader 6f9d13d5dae44a0da997d2540d7c7f1d. Latest consensus state: current_term: 1 leader_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } } }
I20260927 03:28:03.816092 2021 tablet_replica.cc:442] T f70970525c094cecbc8b9d0c341c1f24 P 6fadb3b6c4c04523806c27d488b1cec2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:28:03.816797 2027 tablet_replica.cc:439] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } } }
I20260927 03:28:03.816892 2027 tablet_replica.cc:442] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:28:03.816916 2020 tablet_replica.cc:439] T f70970525c094cecbc8b9d0c341c1f24 P 5803b8bb7755400e8d5c81a565fa588a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } } }
I20260927 03:28:03.816992 2020 tablet_replica.cc:442] T f70970525c094cecbc8b9d0c341c1f24 P 5803b8bb7755400e8d5c81a565fa588a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:28:03.817860 2027 tablet_replica.cc:439] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } } }
I20260927 03:28:03.817946 2027 tablet_replica.cc:442] T f70970525c094cecbc8b9d0c341c1f24 P 6f9d13d5dae44a0da997d2540d7c7f1d: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:28:03.818290 2021 tablet_replica.cc:439] T f70970525c094cecbc8b9d0c341c1f24 P 6fadb3b6c4c04523806c27d488b1cec2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fadb3b6c4c04523806c27d488b1cec2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37057 } } peers { permanent_uuid: "5803b8bb7755400e8d5c81a565fa588a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37755 } } peers { permanent_uuid: "6f9d13d5dae44a0da997d2540d7c7f1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46091 } } }
I20260927 03:28:03.818379 2021 tablet_replica.cc:442] T f70970525c094cecbc8b9d0c341c1f24 P 6fadb3b6c4c04523806c27d488b1cec2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:28:03.821606 2077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:03.819051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57818 (local address 127.0.98.2:37755)
0927 03:28:03.819145 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:03.819147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:03.819170 (+ 23us) server_negotiation.cc:407] Connection header received
0927 03:28:03.819257 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:03.819258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:03.819278 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:03.819351 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:28:03.819504 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:03.819990 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:03.820786 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:03.821005 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:03.821089 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:03.821255 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:28:03.821362 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:28:03.821438 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:03.821501 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 03:28:03.821520 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":52,"threads_started":1}
I20260927 03:28:03.827069 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:03.824288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44717 (local address 127.0.98.3:46091)
0927 03:28:03.824331 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:03.824333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:03.824400 (+ 67us) server_negotiation.cc:407] Connection header received
0927 03:28:03.824487 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:03.824488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:03.824498 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:03.824558 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:28:03.824716 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:03.825312 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:03.826437 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:03.826616 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:03.826663 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:03.826880 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:28:03.826903 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:28:03.826937 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:03.827000 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 03:28:03.827017 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:28:03.827121 2080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:03.824206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46091 (local address 127.0.98.2:44717)
0927 03:28:03.824342 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:03.824353 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:03.824431 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:03.824603 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:03.824604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:03.824606 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:28:03.824646 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:03.824648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:03.825401 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:03.825402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:03.826347 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:03.826350 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:03.826697 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:03.826699 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:03.826765 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:03.826781 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:28:03.826945 (+ 164us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:28:03.826946 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 03:28:03.826998 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 03:28:03.827042 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":84,"threads_started":1}
I20260927 03:28:03.838433 2077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:03.835613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46559 (local address 127.0.98.2:37755)
0927 03:28:03.835648 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:03.835650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:03.835949 (+ 299us) server_negotiation.cc:407] Connection header received
0927 03:28:03.835974 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:03.835975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:03.835983 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:03.836047 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:28:03.836164 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:03.836657 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:03.837558 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:03.837876 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:03.837960 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:03.838181 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:28:03.838220 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:28:03.838277 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:03.838323 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 03:28:03.838349 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 03:28:03.838557 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:03.835583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37755 (local address 127.0.98.3:46559)
0927 03:28:03.835737 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:03.835747 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:03.835820 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:03.836049 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:03.836050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:03.836051 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:28:03.836103 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:03.836104 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:03.836740 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:03.836741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:03.837492 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:03.837495 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:03.837955 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:03.837956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:03.838062 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:03.838081 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:28:03.838280 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:28:03.838282 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:28:03.838317 (+ 35us) client_negotiation.cc:240] Negotiation successful
0927 03:28:03.838382 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":88,"threads_started":1}
I20260927 03:28:03.849107 1508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:28:03.849236 1508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:28:03.849898 1508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc115db5035545bdb6cc4d97816cf042: Sending DeleteTablet for 3 replicas of tablet 00f61fbe663e4356830533b4871ceede
I20260927 03:28:03.850165 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 00f61fbe663e4356830533b4871ceede with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57814
I20260927 03:28:03.850178 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet 00f61fbe663e4356830533b4871ceede with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44088
I20260927 03:28:03.850366 2088 tablet_replica.cc:330] T 00f61fbe663e4356830533b4871ceede P 6fadb3b6c4c04523806c27d488b1cec2: stopping tablet replica
I20260927 03:28:03.850430 2088 raft_consensus.cc:2229] T 00f61fbe663e4356830533b4871ceede P 6fadb3b6c4c04523806c27d488b1cec2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:28:03.850486 2088 raft_consensus.cc:2258] T 00f61fbe663e4356830533b4871ceede P 6fadb3b6c4c04523806c27d488b1cec2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:03.850768 2088 ts_tablet_manager.cc:1905] T 00f61fbe663e4356830533b4871ceede P 6fadb3b6c4c04523806c27d488b1cec2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:03.851056 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 00f61fbe663e4356830533b4871ceede with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45416
I20260927 03:28:03.851563 2090 tablet_replica.cc:330] T 00f61fbe663e4356830533b4871ceede P 6f9d13d5dae44a0da997d2540d7c7f1d: stopping tablet replica
I20260927 03:28:03.851630 2090 raft_consensus.cc:2229] T 00f61fbe663e4356830533b4871ceede P 6f9d13d5dae44a0da997d2540d7c7f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:28:03.851665 2090 raft_consensus.cc:2258] T 00f61fbe663e4356830533b4871ceede P 6f9d13d5dae44a0da997d2540d7c7f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:03.852321 2090 ts_tablet_manager.cc:1905] T 00f61fbe663e4356830533b4871ceede P 6f9d13d5dae44a0da997d2540d7c7f1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:03.853981 2087 tablet_replica.cc:330] T 00f61fbe663e4356830533b4871ceede P 5803b8bb7755400e8d5c81a565fa588a: stopping tablet replica
I20260927 03:28:03.854056 2087 raft_consensus.cc:2229] T 00f61fbe663e4356830533b4871ceede P 5803b8bb7755400e8d5c81a565fa588a [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:28:03.854168 2087 raft_consensus.cc:2258] T 00f61fbe663e4356830533b4871ceede P 5803b8bb7755400e8d5c81a565fa588a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:03.854396 2087 ts_tablet_manager.cc:1905] T 00f61fbe663e4356830533b4871ceede P 5803b8bb7755400e8d5c81a565fa588a: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 03:28:03 dist-test-slave-j0hf krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479682, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 03:28:03.869494 2091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:03.851405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44092 (local address 127.0.98.1:37057)
0927 03:28:03.855513 (+ 4108us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:03.855515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:03.859674 (+ 4159us) server_negotiation.cc:407] Connection header received
0927 03:28:03.859721 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:03.859722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:03.859736 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:03.859840 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:03.860545 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:03.861349 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:03.862408 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:03.862738 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:03.862830 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:03.865939 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:03.865955 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:03.865956 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:03.865965 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:03.868150 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:03.868576 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:03.868580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:03.868581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:03.868649 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:03.868866 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:03.868867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:03.868868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:03.869062 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:03.869206 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:03.869286 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 03:28:03.869374 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4085,"thread_start_us":87,"threads_started":1}
W20260927 03:28:03.869850 1705 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44092
I20260927 03:28:03.871614 2088 ts_tablet_manager.cc:1918] T 00f61fbe663e4356830533b4871ceede P 6fadb3b6c4c04523806c27d488b1cec2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:28:03.871707 2088 log.cc:1196] T 00f61fbe663e4356830533b4871ceede P 6fadb3b6c4c04523806c27d488b1cec2: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-0/wal/wals/00f61fbe663e4356830533b4871ceede
I20260927 03:28:03.872874 2090 ts_tablet_manager.cc:1918] T 00f61fbe663e4356830533b4871ceede P 6f9d13d5dae44a0da997d2540d7c7f1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:28:03.872949 2090 log.cc:1196] T 00f61fbe663e4356830533b4871ceede P 6f9d13d5dae44a0da997d2540d7c7f1d: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-2/wal/wals/00f61fbe663e4356830533b4871ceede
I20260927 03:28:03.873202 2090 ts_tablet_manager.cc:1939] T 00f61fbe663e4356830533b4871ceede P 6f9d13d5dae44a0da997d2540d7c7f1d: Deleting consensus metadata
I20260927 03:28:03.873598 1493 catalog_manager.cc:4841] TS 6f9d13d5dae44a0da997d2540d7c7f1d (127.0.98.3:46091): tablet 00f61fbe663e4356830533b4871ceede (table test-table [id=0bb849292c81455b8f5b2b0b07c11388]) successfully deleted
I20260927 03:28:03.873826 2087 ts_tablet_manager.cc:1918] T 00f61fbe663e4356830533b4871ceede P 5803b8bb7755400e8d5c81a565fa588a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:28:03.874300 2087 log.cc:1196] T 00f61fbe663e4356830533b4871ceede P 5803b8bb7755400e8d5c81a565fa588a: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790479675865103-392-0/minicluster-data/ts-1/wal/wals/00f61fbe663e4356830533b4871ceede
I20260927 03:28:03.874588 2087 ts_tablet_manager.cc:1939] T 00f61fbe663e4356830533b4871ceede P 5803b8bb7755400e8d5c81a565fa588a: Deleting consensus metadata
I20260927 03:28:03.874977 1494 catalog_manager.cc:4841] TS 5803b8bb7755400e8d5c81a565fa588a (127.0.98.2:37755): tablet 00f61fbe663e4356830533b4871ceede (table test-table [id=0bb849292c81455b8f5b2b0b07c11388]) successfully deleted
I20260927 03:28:03.875168 2088 ts_tablet_manager.cc:1939] T 00f61fbe663e4356830533b4871ceede P 6fadb3b6c4c04523806c27d488b1cec2: Deleting consensus metadata
I20260927 03:28:03.875603 1492 catalog_manager.cc:4841] TS 6fadb3b6c4c04523806c27d488b1cec2 (127.0.98.1:37057): tablet 00f61fbe663e4356830533b4871ceede (table test-table [id=0bb849292c81455b8f5b2b0b07c11388]) successfully deleted
I20260927 03:28:03.887804 2094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:03.879746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49972 (local address 127.0.98.62:40781)
0927 03:28:03.879864 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:03.879866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:03.880061 (+ 195us) server_negotiation.cc:407] Connection header received
0927 03:28:03.880121 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:03.880122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:03.880134 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:03.880202 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:03.880959 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:03.881776 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:03.882895 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:03.883182 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:03.883232 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:03.884785 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:03.884787 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:03.884787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:03.884795 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:03.886452 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:03.886865 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:03.886866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:03.886866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:03.886929 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:03.887126 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:03.887127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:03.887127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:03.887257 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:03.887341 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:03.887591 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 03:28:03.887673 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":57,"threads_started":1}
W20260927 03:28:03.888073 1508 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49972
I20260927 03:28:03.888654 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 1580
I20260927 03:28:03.895174 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 1720
I20260927 03:28:03.901325 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 1860
I20260927 03:28:03.909627 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 1476
2026-09-27T03:28:03Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3000 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:28:03 dist-test-slave-j0hf krb5kdc[2098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:28:03 dist-test-slave-j0hf krb5kdc[2098](info): set up 2 sockets
Sep 27 03:28:03 dist-test-slave-j0hf krb5kdc[2098](info): commencing operation
krb5kdc: starting...
W20260927 03:28:05.003183 392 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:28:05 dist-test-slave-j0hf krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:28:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:28:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:28:05.130295 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42161
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:05.214687 2111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:05.214963 2111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:05.215006 2111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:05.216223 2111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:28:05.216311 2111 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 03:28:05.216343 2111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:05.216374 2111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:05.216400 2111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:28:05.216434 2111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:28:05.217936 2111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42161
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:05.218221 2111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:05.218458 2111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:05.223480 2117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:05.223471 2119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:05.223500 2111 server_base.cc:1034] running on GCE node
W20260927 03:28:05.223471 2116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:05.223963 2111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:05.224308 2111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:05.225502 2111 hybrid_clock.cc:648] HybridClock initialized: now 1790479685225512 us; error 56 us; skew 500 ppm
Sep 27 03:28:05 dist-test-slave-j0hf krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:05.228087 2111 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:28:05.228567 2111 webserver.cc:460] Webserver started at http://127.0.98.62:34535/ using document root <none> and password file <none>
I20260927 03:28:05.228770 2111 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:05.228818 2111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:05.228909 2111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:05.229928 2111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "80fc0bdfbbfc4e259a2ac9cb3841e2d5"
format_stamp: "Formatted at 2026-09-27 03:28:05 on dist-test-slave-j0hf"
server_key: "e849d52ff307fd0d8ad6025f02774124"
server_key_iv: "37e3dfcfa41e1278598674264b557767"
server_key_version: "encryptionkey@0"
I20260927 03:28:05.230232 2111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "80fc0bdfbbfc4e259a2ac9cb3841e2d5"
format_stamp: "Formatted at 2026-09-27 03:28:05 on dist-test-slave-j0hf"
server_key: "e849d52ff307fd0d8ad6025f02774124"
server_key_iv: "37e3dfcfa41e1278598674264b557767"
server_key_version: "encryptionkey@0"
I20260927 03:28:05.231546 2111 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:28:05.232365 2126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:05.232623 2111 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:05.232731 2111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "80fc0bdfbbfc4e259a2ac9cb3841e2d5"
format_stamp: "Formatted at 2026-09-27 03:28:05 on dist-test-slave-j0hf"
server_key: "e849d52ff307fd0d8ad6025f02774124"
server_key_iv: "37e3dfcfa41e1278598674264b557767"
server_key_version: "encryptionkey@0"
I20260927 03:28:05.232828 2111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:05.258949 2111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:05.261441 2111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:05.261624 2111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:05.266538 2111 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42161
I20260927 03:28:05.266584 2188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42161 every 8 connection(s)
I20260927 03:28:05.266929 2111 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:28:05.267601 2189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:05.270002 2189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80fc0bdfbbfc4e259a2ac9cb3841e2d5: Bootstrap starting.
I20260927 03:28:05.270632 2189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80fc0bdfbbfc4e259a2ac9cb3841e2d5: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:05.270987 2189 log.cc:824] T 00000000000000000000000000000000 P 80fc0bdfbbfc4e259a2ac9cb3841e2d5: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:05.271075 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 2111
I20260927 03:28:05.271152 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:28:05.271328 392 external_mini_cluster.cc:1428] Setting key c263ff05d92dd727a0fc2875285d6b0e
I20260927 03:28:05.271924 2189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80fc0bdfbbfc4e259a2ac9cb3841e2d5: No bootstrap required, opened a new log
I20260927 03:28:05.274112 2189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80fc0bdfbbfc4e259a2ac9cb3841e2d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80fc0bdfbbfc4e259a2ac9cb3841e2d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42161 } }
I20260927 03:28:05.274262 2189 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80fc0bdfbbfc4e259a2ac9cb3841e2d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:05.274286 2189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80fc0bdfbbfc4e259a2ac9cb3841e2d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fc0bdfbbfc4e259a2ac9cb3841e2d5, State: Initialized, Role: FOLLOWER
I20260927 03:28:05.274438 2189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80fc0bdfbbfc4e259a2ac9cb3841e2d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80fc0bdfbbfc4e259a2ac9cb3841e2d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42161 } }
I20260927 03:28:05.274540 2189 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80fc0bdfbbfc4e259a2ac9cb3841e2d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:28:05.274575 2189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80fc0bdfbbfc4e259a2ac9cb3841e2d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:28:05.274628 2189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80fc0bdfbbfc4e259a2ac9cb3841e2d5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:05.275391 2189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80fc0bdfbbfc4e259a2ac9cb3841e2d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80fc0bdfbbfc4e259a2ac9cb3841e2d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42161 } }
I20260927 03:28:05.275568 2189 leader_election.cc:302] T 00000000000000000000000000000000 P 80fc0bdfbbfc4e259a2ac9cb3841e2d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80fc0bdfbbfc4e259a2ac9cb3841e2d5; no voters:
I20260927 03:28:05.275770 2189 leader_election.cc:288] T 00000000000000000000000000000000 P 80fc0bdfbbfc4e259a2ac9cb3841e2d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:28:05.275825 2194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80fc0bdfbbfc4e259a2ac9cb3841e2d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:05.276041 2194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80fc0bdfbbfc4e259a2ac9cb3841e2d5 [term 1 LEADER]: Becoming Leader. State: Replica: 80fc0bdfbbfc4e259a2ac9cb3841e2d5, State: Running, Role: LEADER
I20260927 03:28:05.276052 2189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80fc0bdfbbfc4e259a2ac9cb3841e2d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:28:05.276208 2194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80fc0bdfbbfc4e259a2ac9cb3841e2d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80fc0bdfbbfc4e259a2ac9cb3841e2d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42161 } }
I20260927 03:28:05.276654 2195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80fc0bdfbbfc4e259a2ac9cb3841e2d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80fc0bdfbbfc4e259a2ac9cb3841e2d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80fc0bdfbbfc4e259a2ac9cb3841e2d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42161 } } }
I20260927 03:28:05.276742 2195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80fc0bdfbbfc4e259a2ac9cb3841e2d5 [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:05.276902 2196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80fc0bdfbbfc4e259a2ac9cb3841e2d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80fc0bdfbbfc4e259a2ac9cb3841e2d5. Latest consensus state: current_term: 1 leader_uuid: "80fc0bdfbbfc4e259a2ac9cb3841e2d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80fc0bdfbbfc4e259a2ac9cb3841e2d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42161 } } }
I20260927 03:28:05.276944 2198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:28:05.276965 2196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80fc0bdfbbfc4e259a2ac9cb3841e2d5 [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:05.277503 2198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:28:05.279114 2198 catalog_manager.cc:1347] Generated new cluster ID: 206e4273471b40a88ba5108fb57c3959
I20260927 03:28:05.279165 2198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:28:05 dist-test-slave-j0hf krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:05.285398 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:05.271914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40140 (local address 127.0.98.62:42161)
0927 03:28:05.272123 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:05.272133 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:05.272168 (+ 35us) server_negotiation.cc:407] Connection header received
0927 03:28:05.274110 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:05.274118 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:05.274179 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:05.274376 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:05.275185 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.276568 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.277646 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.278270 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.278413 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:05.281384 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:05.281390 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:05.281390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:05.281404 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:05.283554 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:05.283981 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:05.283985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:05.283986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:05.284177 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:05.284400 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:05.284401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:05.284401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:05.284756 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:05.284927 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:05.285018 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 03:28:05.285173 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":97,"threads_started":1}
I20260927 03:28:05.287376 2198 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:28:05.287925 2198 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:28:05.300376 2198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80fc0bdfbbfc4e259a2ac9cb3841e2d5: Generated new TSK 0
I20260927 03:28:05.300632 2198 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:28:05.331455 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42161
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:05.416620 2215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:05.416841 2215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:05.416868 2215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:05.418145 2215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:05.418193 2215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:05.418215 2215 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 03:28:05.418268 2215 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:28:05.418296 2215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:28:05.419873 2215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:05.420149 2215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:05.420398 2215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:05.425594 2223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:05.425693 2221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:05.425676 2215 server_base.cc:1034] running on GCE node
W20260927 03:28:05.425594 2220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:05.426056 2215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:05.426399 2215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:05.427578 2215 hybrid_clock.cc:648] HybridClock initialized: now 1790479685427559 us; error 39 us; skew 500 ppm
Sep 27 03:28:05 dist-test-slave-j0hf krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:05.430224 2215 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:28:05.430712 2215 webserver.cc:460] Webserver started at http://127.0.98.1:33837/ using document root <none> and password file <none>
I20260927 03:28:05.430967 2215 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:05.431037 2215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:05.431190 2215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:05.432345 2215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-0/data/instance:
uuid: "5d9947006e7c4ae89d95618bb8ed6a48"
format_stamp: "Formatted at 2026-09-27 03:28:05 on dist-test-slave-j0hf"
server_key: "c10e798403be8d92941bec292ec41c37"
server_key_iv: "787244e2f43292602edcf0e21676092e"
server_key_version: "encryptionkey@0"
I20260927 03:28:05.432711 2215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5d9947006e7c4ae89d95618bb8ed6a48"
format_stamp: "Formatted at 2026-09-27 03:28:05 on dist-test-slave-j0hf"
server_key: "c10e798403be8d92941bec292ec41c37"
server_key_iv: "787244e2f43292602edcf0e21676092e"
server_key_version: "encryptionkey@0"
I20260927 03:28:05.434109 2215 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:28:05.435078 2230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:05.435415 2215 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:05.435576 2215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "5d9947006e7c4ae89d95618bb8ed6a48"
format_stamp: "Formatted at 2026-09-27 03:28:05 on dist-test-slave-j0hf"
server_key: "c10e798403be8d92941bec292ec41c37"
server_key_iv: "787244e2f43292602edcf0e21676092e"
server_key_version: "encryptionkey@0"
I20260927 03:28:05.435654 2215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:05.452661 2215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:05.454638 2215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:05.454754 2215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:05.455228 2215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:05.455307 2215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:05.455358 2215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:05.455389 2215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:05.462680 2215 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41697
I20260927 03:28:05.462769 2345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41697 every 8 connection(s)
I20260927 03:28:05.463140 2215 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:28:05 dist-test-slave-j0hf krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:05.466476 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:05.455802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40329 (local address 127.0.98.62:42161)
0927 03:28:05.455842 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:05.455844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:05.456102 (+ 258us) server_negotiation.cc:407] Connection header received
0927 03:28:05.456648 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:05.456649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:05.456665 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:05.456716 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:05.457631 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.458503 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.460240 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.460664 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.460769 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:05.463819 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:05.463820 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:05.463821 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:05.463834 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:05.465210 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:05.465627 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:05.465628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:05.465629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:05.465683 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:05.465904 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:05.465904 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:05.465905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:05.466024 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:05.466077 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:05.466318 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 03:28:05.466391 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:28:05.466629 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:05.455810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42161 (local address 127.0.98.1:40329)
0927 03:28:05.455985 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:05.456020 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:05.456535 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:05.456789 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:05.456792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:05.457193 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:05.457507 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:05.457514 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.458618 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.458619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:05.460133 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:05.460136 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.460803 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.460804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:05.460941 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:05.461714 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:05.461726 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:05.463695 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:05.465319 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:05.465325 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:05.465327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:05.465557 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:05.465741 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:05.465742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:05.465742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:05.465845 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:05.466083 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:05.466087 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:05.466207 (+ 120us) client_negotiation.cc:769] Sending connection context
0927 03:28:05.466290 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 03:28:05.466416 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
I20260927 03:28:05.467067 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 2215
I20260927 03:28:05.467164 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:28:05.467195 2346 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42161
I20260927 03:28:05.467278 2346 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:05.467381 392 external_mini_cluster.cc:1428] Setting key eb2453ae2994a7b8be31c60304ee361d
I20260927 03:28:05.467535 2346 heartbeater.cc:511] Master 127.0.98.62:42161 requested a full tablet report, sending...
I20260927 03:28:05.468302 2143 ts_manager.cc:194] Registered new tserver with Master: 5d9947006e7c4ae89d95618bb8ed6a48 (127.0.98.1:41697)
I20260927 03:28:05.469208 2143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40329
I20260927 03:28:05.476034 2353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:05.469567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42161 (local address 127.0.98.1:60453)
0927 03:28:05.469719 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:05.469732 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:05.469809 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:05.469929 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:05.469929 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:05.470183 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:05.470254 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:05.470257 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.471100 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.471101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:05.471674 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:05.471675 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.472181 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.472182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:05.472242 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:05.472733 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:05.472737 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:05.473032 (+ 295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:05.474897 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:05.474898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:05.474898 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:05.475089 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:05.475304 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:05.475305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:05.475305 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:05.475369 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:05.475687 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:05.475688 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:05.475729 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 03:28:05.475851 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 03:28:05.475944 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260927 03:28:05.476078 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:05.469630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60453 (local address 127.0.98.62:42161)
0927 03:28:05.469711 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:05.469713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:05.469782 (+ 69us) server_negotiation.cc:407] Connection header received
0927 03:28:05.469858 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:05.469859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:05.469870 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:05.469924 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:05.470347 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.471008 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.471752 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.472095 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.472166 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:05.473121 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:05.473123 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:05.473123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:05.473131 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:05.474794 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:05.475175 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:05.475176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:05.475176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:05.475234 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:05.475463 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:05.475464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:05.475464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:05.475614 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:05.475690 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:05.475885 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 03:28:05.475976 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:28:05.494407 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42161
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:05.576717 2355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:05.576964 2355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:05.576993 2355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:05.578262 2355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:05.578305 2355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:05.578326 2355 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 03:28:05.578338 2355 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:28:05.578356 2355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:28:05.579768 2355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:05.579957 2355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:05.580134 2355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:05.585222 2363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:05.585263 2355 server_base.cc:1034] running on GCE node
W20260927 03:28:05.585230 2360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:05.585229 2361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:05.585596 2355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:05.585862 2355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:05.587042 2355 hybrid_clock.cc:648] HybridClock initialized: now 1790479685587021 us; error 35 us; skew 500 ppm
Sep 27 03:28:05 dist-test-slave-j0hf krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:05.589613 2355 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:28:05.590129 2355 webserver.cc:460] Webserver started at http://127.0.98.2:33635/ using document root <none> and password file <none>
I20260927 03:28:05.590385 2355 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:05.590462 2355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:05.590622 2355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:05.591764 2355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-1/data/instance:
uuid: "74c74bf308644ca1a7c3d6715a1d504f"
format_stamp: "Formatted at 2026-09-27 03:28:05 on dist-test-slave-j0hf"
server_key: "b738e52e704f90ccb9d93ff00beca27e"
server_key_iv: "2ad1801568541fd72575717101a269b2"
server_key_version: "encryptionkey@0"
I20260927 03:28:05.592129 2355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-1/wal/instance:
uuid: "74c74bf308644ca1a7c3d6715a1d504f"
format_stamp: "Formatted at 2026-09-27 03:28:05 on dist-test-slave-j0hf"
server_key: "b738e52e704f90ccb9d93ff00beca27e"
server_key_iv: "2ad1801568541fd72575717101a269b2"
server_key_version: "encryptionkey@0"
I20260927 03:28:05.593427 2355 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:05.594235 2370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:05.594480 2355 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:05.594581 2355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "74c74bf308644ca1a7c3d6715a1d504f"
format_stamp: "Formatted at 2026-09-27 03:28:05 on dist-test-slave-j0hf"
server_key: "b738e52e704f90ccb9d93ff00beca27e"
server_key_iv: "2ad1801568541fd72575717101a269b2"
server_key_version: "encryptionkey@0"
I20260927 03:28:05.594663 2355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:05.608359 2355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:05.610304 2355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:05.610481 2355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:05.611054 2355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:05.611114 2355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:05.611157 2355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:05.611179 2355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 03:28:05 dist-test-slave-j0hf krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:05.618639 2355 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42071
I20260927 03:28:05.618830 2485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42071 every 8 connection(s)
I20260927 03:28:05.619071 2355 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:28:05.619359 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 2355
I20260927 03:28:05.619483 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:28:05.619700 392 external_mini_cluster.cc:1428] Setting key 9d12cf045a65bae693f315da21c68854
I20260927 03:28:05.620594 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:05.611504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56021 (local address 127.0.98.62:42161)
0927 03:28:05.611580 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:05.611582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:05.611796 (+ 214us) server_negotiation.cc:407] Connection header received
0927 03:28:05.612263 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:05.612265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:05.612278 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:05.612364 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:05.613002 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.613574 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.614597 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.614806 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.614885 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:05.617390 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:05.617392 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:05.617392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:05.617400 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:05.619000 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:05.619538 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:05.619539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:05.619539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:05.619602 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:05.619905 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:05.619906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:05.619906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:05.620076 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:05.620142 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:05.620461 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 03:28:05.620529 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 03:28:05.620859 2379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:05.611487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42161 (local address 127.0.98.2:56021)
0927 03:28:05.611653 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:05.611693 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:05.612162 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:05.612414 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:05.612417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:05.612656 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:05.612900 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:05.612905 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.613677 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.613678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:05.614503 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:05.614505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.614889 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.614891 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:05.615011 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:05.615552 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:05.615560 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:05.617294 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:05.619095 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:05.619101 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:05.619103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:05.619444 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:05.619669 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:05.619670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:05.619671 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:05.619810 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:05.620169 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:05.620173 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:05.620294 (+ 121us) client_negotiation.cc:769] Sending connection context
0927 03:28:05.620410 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 03:28:05.620590 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
I20260927 03:28:05.621312 2486 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42161
I20260927 03:28:05.621392 2486 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:05.621579 2486 heartbeater.cc:511] Master 127.0.98.62:42161 requested a full tablet report, sending...
I20260927 03:28:05.621907 2143 ts_manager.cc:194] Registered new tserver with Master: 74c74bf308644ca1a7c3d6715a1d504f (127.0.98.2:42071)
I20260927 03:28:05.622262 2143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56021
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 03:28:05.629812 2493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:05.623634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42161 (local address 127.0.98.2:56479)
0927 03:28:05.623776 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:05.623788 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:05.623847 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:05.623962 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:05.623962 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:05.624176 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:05.624243 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:05.624244 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.624866 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.624867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:05.625806 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:05.625809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.626135 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.626136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:05.626206 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:05.626607 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:05.626612 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:05.627032 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:05.628778 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:05.628779 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:05.628780 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:05.629019 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:05.629219 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:05.629220 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:05.629220 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:05.629273 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:05.629501 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:05.629501 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:05.629537 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 03:28:05.629586 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 03:28:05.629701 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260927 03:28:05.629837 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:05.623717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56479 (local address 127.0.98.62:42161)
0927 03:28:05.623762 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:05.623764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:05.623835 (+ 71us) server_negotiation.cc:407] Connection header received
0927 03:28:05.623900 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:05.623900 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:05.623907 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:05.623950 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:05.624317 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.624769 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.625896 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.626066 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.626116 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:05.627141 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:05.627143 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:05.627143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:05.627159 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:05.628647 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:05.629096 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:05.629097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:05.629098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:05.629149 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:05.629314 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:05.629314 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:05.629314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:05.629447 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:05.629498 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:05.629601 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 03:28:05.629719 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:28:05.648499 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42161
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:05.731314 2495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:05.731597 2495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:05.731626 2495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:05.732920 2495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:05.732965 2495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:05.732986 2495 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 03:28:05.732998 2495 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:28:05.733044 2495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:28:05.734520 2495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:05.734768 2495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:05.735019 2495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:05.740286 2501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:05.740312 2500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:05.740294 2503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:05.740599 2495 server_base.cc:1034] running on GCE node
I20260927 03:28:05.740804 2495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:05.741053 2495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:05.742234 2495 hybrid_clock.cc:648] HybridClock initialized: now 1790479685742217 us; error 28 us; skew 500 ppm
Sep 27 03:28:05 dist-test-slave-j0hf krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:05.744913 2495 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:28:05.745420 2495 webserver.cc:460] Webserver started at http://127.0.98.3:35449/ using document root <none> and password file <none>
I20260927 03:28:05.745703 2495 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:05.745776 2495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:05.745939 2495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:05.747026 2495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-2/data/instance:
uuid: "c16c0dcf19b2426e8d54a191b46cf973"
format_stamp: "Formatted at 2026-09-27 03:28:05 on dist-test-slave-j0hf"
server_key: "7258929422d01c6c25d2e601bd5a8b93"
server_key_iv: "eef0774204c824d491ef826f31927884"
server_key_version: "encryptionkey@0"
I20260927 03:28:05.747392 2495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c16c0dcf19b2426e8d54a191b46cf973"
format_stamp: "Formatted at 2026-09-27 03:28:05 on dist-test-slave-j0hf"
server_key: "7258929422d01c6c25d2e601bd5a8b93"
server_key_iv: "eef0774204c824d491ef826f31927884"
server_key_version: "encryptionkey@0"
I20260927 03:28:05.748826 2495 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:05.749552 2510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:05.749718 2495 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:05.749817 2495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "c16c0dcf19b2426e8d54a191b46cf973"
format_stamp: "Formatted at 2026-09-27 03:28:05 on dist-test-slave-j0hf"
server_key: "7258929422d01c6c25d2e601bd5a8b93"
server_key_iv: "eef0774204c824d491ef826f31927884"
server_key_version: "encryptionkey@0"
I20260927 03:28:05.749905 2495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:05.760046 2495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:05.762015 2495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:05.762187 2495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:05.762805 2495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:05.762856 2495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:05.762887 2495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:05.762907 2495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 03:28:05 dist-test-slave-j0hf krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:05.770473 2495 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42561
I20260927 03:28:05.770602 2625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42561 every 8 connection(s)
I20260927 03:28:05.770893 2495 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:28:05.773588 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 2495
I20260927 03:28:05.773682 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:28:05.773761 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:05.763270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57167 (local address 127.0.98.62:42161)
0927 03:28:05.763319 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:05.763321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:05.763704 (+ 383us) server_negotiation.cc:407] Connection header received
0927 03:28:05.764460 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:05.764462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:05.764476 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:05.764542 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:05.765503 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.766003 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.767717 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.767997 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.768088 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:05.770793 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:05.770794 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:05.770795 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:05.770803 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:05.772260 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:05.772657 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:05.772658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:05.772658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:05.772717 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:05.773030 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:05.773031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:05.773031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:05.773143 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:05.773197 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:05.773596 (+ 399us) server_negotiation.cc:299] Negotiation successful
0927 03:28:05.773697 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:28:05.773883 392 external_mini_cluster.cc:1428] Setting key 5872b8be08fa36460ff8cc2b9770a1b9
I20260927 03:28:05.773972 2519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:05.763258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42161 (local address 127.0.98.3:57167)
0927 03:28:05.763518 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:05.763579 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:05.764326 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:05.764629 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:05.764634 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:05.765071 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:05.765379 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:05.765386 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.766077 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.766078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:05.767629 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:05.767632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.768109 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.768110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:05.768292 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:05.768822 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:05.768835 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:05.770681 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:05.772337 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:05.772342 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:05.772345 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:05.772567 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:05.772846 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:05.772847 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:05.772848 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:05.772963 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:05.773188 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:05.773191 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:05.773307 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 03:28:05.773537 (+ 230us) client_negotiation.cc:240] Negotiation successful
0927 03:28:05.773739 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":192,"threads_started":1}
I20260927 03:28:05.775296 2626 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42161
I20260927 03:28:05.775360 2626 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:05.775602 2626 heartbeater.cc:511] Master 127.0.98.62:42161 requested a full tablet report, sending...
I20260927 03:28:05.776023 2143 ts_manager.cc:194] Registered new tserver with Master: c16c0dcf19b2426e8d54a191b46cf973 (127.0.98.3:42561)
I20260927 03:28:05.776412 2143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57167
I20260927 03:28:05.781283 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:05.775400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42161 (local address 127.0.98.3:43005)
0927 03:28:05.775591 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:05.775603 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:05.775665 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:05.775801 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:05.775802 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:05.776003 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:05.776075 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:05.776077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.776921 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.776922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:05.777685 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:05.777687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.777995 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.777995 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:05.778037 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:05.778697 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:05.778702 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:05.779008 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:05.780469 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:05.780470 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:05.780470 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:05.780625 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:05.780791 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:05.780792 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:05.780792 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:05.780840 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:05.781050 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:05.781050 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:05.781082 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 03:28:05.781142 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 03:28:05.781213 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":52,"thread_start_us":122,"threads_started":1}
I20260927 03:28:05.781284 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:05.775518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43005 (local address 127.0.98.62:42161)
0927 03:28:05.775561 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:05.775563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:05.775652 (+ 89us) server_negotiation.cc:407] Connection header received
0927 03:28:05.775715 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:05.775716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:05.775726 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:05.775784 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:05.776167 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.776827 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.777755 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.777933 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.777991 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:05.779102 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:05.779103 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:05.779104 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:05.779109 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:05.780400 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:05.780689 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:05.780689 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:05.780690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:05.780736 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:05.780891 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:05.780892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:05.780893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:05.780997 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:05.781046 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:05.781172 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 03:28:05.781226 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:28:05.785573 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 03:28:05 dist-test-slave-j0hf krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 03:28:05 dist-test-slave-j0hf krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:05.807853 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:05.800921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40156 (local address 127.0.98.62:42161)
0927 03:28:05.801007 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:05.801009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:05.801032 (+ 23us) server_negotiation.cc:407] Connection header received
0927 03:28:05.801063 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:05.801064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:05.801072 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:05.801133 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:05.801712 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.802210 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.802984 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.803162 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.803248 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:05.805446 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:05.805447 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:05.805448 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:05.805457 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:05.806670 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:05.807022 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:05.807023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:05.807024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:05.807081 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:05.807275 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:05.807276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:05.807276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:05.807392 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:05.807545 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:05.807663 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 03:28:05.807746 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 03:28:05.809345 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:28:05.810847 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:28:05.817605 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:05.813752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39962 (local address 127.0.98.3:42561)
0927 03:28:05.813974 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:05.813978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:05.813999 (+ 21us) server_negotiation.cc:407] Connection header received
0927 03:28:05.814037 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:05.814039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:05.814054 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:05.814124 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:05.814314 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.815319 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.816443 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.817142 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.817221 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:05.817283 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:05.817452 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 03:28:05.817497 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":81,"threads_started":1}
I20260927 03:28:05.817637 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:05.813143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42071 (local address 127.0.0.1:58852)
0927 03:28:05.813260 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:05.813305 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:05.813372 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:05.813726 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:05.813729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:05.813732 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:05.813808 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:05.813810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.815197 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.815198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:05.816299 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:05.816301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.817236 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.817236 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:05.817341 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:05.817351 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 03:28:05.817428 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 03:28:05.817516 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":68,"thread_start_us":69,"threads_started":1}
I20260927 03:28:05.817636 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:05.813340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42561 (local address 127.0.0.1:39962)
0927 03:28:05.813573 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:05.813583 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:05.813627 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:05.814189 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:05.814190 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:05.814192 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:05.814233 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:05.814234 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.815398 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.815399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:05.816332 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:05.816334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.817234 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.817235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:05.817336 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:05.817351 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 03:28:05.817428 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 03:28:05.817516 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":7,"thread_start_us":67,"threads_started":1}
I20260927 03:28:05.817902 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:05.813365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58852 (local address 127.0.98.2:42071)
0927 03:28:05.813541 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:05.813546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:05.813557 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:28:05.813602 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:05.813605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:05.813621 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:05.813895 (+ 274us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:05.813973 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.815109 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.816390 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.817149 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.817221 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:05.817283 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:05.817764 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 03:28:05.817809 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":113,"threads_started":1}
I20260927 03:28:05.818655 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:05.813183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41697 (local address 127.0.0.1:42052)
0927 03:28:05.813326 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:05.813337 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:05.813387 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:05.813905 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:05.813905 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:05.813908 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:05.813965 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:05.813967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.814967 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.814968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:05.816299 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:05.816301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.818413 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.818414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:05.818487 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:05.818491 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:28:05.818550 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 03:28:05.818590 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":5,"thread_start_us":101,"threads_started":1}
I20260927 03:28:05.818526 2550 tablet_service.cc:1467] Processing CreateTablet for tablet a5083f810bf94b3e9824825f4abb7a31 (DEFAULT_TABLE table=test-table [id=ead9a8b1e77a41bfaacf58f20be90e2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:05.818514 2420 tablet_service.cc:1467] Processing CreateTablet for tablet a5083f810bf94b3e9824825f4abb7a31 (DEFAULT_TABLE table=test-table [id=ead9a8b1e77a41bfaacf58f20be90e2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:05.818704 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:05.813486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42052 (local address 127.0.98.1:41697)
0927 03:28:05.813650 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:05.813655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:05.813666 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:28:05.813707 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:05.813709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:05.813724 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:05.814066 (+ 342us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:05.814133 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.814874 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.817323 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.818314 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.818407 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:05.818477 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:05.818573 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 03:28:05.818613 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
I20260927 03:28:05.818893 2550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5083f810bf94b3e9824825f4abb7a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:05.818898 2420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5083f810bf94b3e9824825f4abb7a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:05.819357 2280 tablet_service.cc:1467] Processing CreateTablet for tablet a5083f810bf94b3e9824825f4abb7a31 (DEFAULT_TABLE table=test-table [id=ead9a8b1e77a41bfaacf58f20be90e2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:05.819741 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5083f810bf94b3e9824825f4abb7a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:05.821928 2649 tablet_bootstrap.cc:492] T a5083f810bf94b3e9824825f4abb7a31 P c16c0dcf19b2426e8d54a191b46cf973: Bootstrap starting.
I20260927 03:28:05.822695 2650 tablet_bootstrap.cc:492] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f: Bootstrap starting.
I20260927 03:28:05.823051 2649 tablet_bootstrap.cc:654] T a5083f810bf94b3e9824825f4abb7a31 P c16c0dcf19b2426e8d54a191b46cf973: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:05.823323 2650 tablet_bootstrap.cc:654] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:05.823390 2649 log.cc:824] T a5083f810bf94b3e9824825f4abb7a31 P c16c0dcf19b2426e8d54a191b46cf973: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:05.823663 2650 log.cc:824] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:05.823702 2651 tablet_bootstrap.cc:492] T a5083f810bf94b3e9824825f4abb7a31 P 5d9947006e7c4ae89d95618bb8ed6a48: Bootstrap starting.
I20260927 03:28:05.824175 2649 tablet_bootstrap.cc:492] T a5083f810bf94b3e9824825f4abb7a31 P c16c0dcf19b2426e8d54a191b46cf973: No bootstrap required, opened a new log
I20260927 03:28:05.824249 2649 ts_tablet_manager.cc:1397] T a5083f810bf94b3e9824825f4abb7a31 P c16c0dcf19b2426e8d54a191b46cf973: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:28:05.824282 2651 tablet_bootstrap.cc:654] T a5083f810bf94b3e9824825f4abb7a31 P 5d9947006e7c4ae89d95618bb8ed6a48: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:05.824467 2650 tablet_bootstrap.cc:492] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f: No bootstrap required, opened a new log
I20260927 03:28:05.824533 2650 ts_tablet_manager.cc:1397] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:28:05.824615 2651 log.cc:824] T a5083f810bf94b3e9824825f4abb7a31 P 5d9947006e7c4ae89d95618bb8ed6a48: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:05.825268 2651 tablet_bootstrap.cc:492] T a5083f810bf94b3e9824825f4abb7a31 P 5d9947006e7c4ae89d95618bb8ed6a48: No bootstrap required, opened a new log
I20260927 03:28:05.825347 2651 ts_tablet_manager.cc:1397] T a5083f810bf94b3e9824825f4abb7a31 P 5d9947006e7c4ae89d95618bb8ed6a48: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:28:05.826217 2649 raft_consensus.cc:348] T a5083f810bf94b3e9824825f4abb7a31 P c16c0dcf19b2426e8d54a191b46cf973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } } peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } }
I20260927 03:28:05.826395 2649 raft_consensus.cc:374] T a5083f810bf94b3e9824825f4abb7a31 P c16c0dcf19b2426e8d54a191b46cf973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:05.826429 2649 raft_consensus.cc:729] T a5083f810bf94b3e9824825f4abb7a31 P c16c0dcf19b2426e8d54a191b46cf973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c16c0dcf19b2426e8d54a191b46cf973, State: Initialized, Role: FOLLOWER
I20260927 03:28:05.826380 2650 raft_consensus.cc:348] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } } peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } }
I20260927 03:28:05.826509 2650 raft_consensus.cc:374] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:05.826551 2650 raft_consensus.cc:729] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c74bf308644ca1a7c3d6715a1d504f, State: Initialized, Role: FOLLOWER
I20260927 03:28:05.826584 2649 consensus_queue.cc:260] T a5083f810bf94b3e9824825f4abb7a31 P c16c0dcf19b2426e8d54a191b46cf973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } } peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } }
I20260927 03:28:05.826649 2650 consensus_queue.cc:260] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } } peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } }
I20260927 03:28:05.826864 2649 ts_tablet_manager.cc:1428] T a5083f810bf94b3e9824825f4abb7a31 P c16c0dcf19b2426e8d54a191b46cf973: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:28:05.826889 2650 ts_tablet_manager.cc:1428] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 03:28:05.826982 2626 heartbeater.cc:503] Master 127.0.98.62:42161 was elected leader, sending a full tablet report...
I20260927 03:28:05.827018 2486 heartbeater.cc:503] Master 127.0.98.62:42161 was elected leader, sending a full tablet report...
I20260927 03:28:05.827081 2651 raft_consensus.cc:348] T a5083f810bf94b3e9824825f4abb7a31 P 5d9947006e7c4ae89d95618bb8ed6a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } } peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } }
I20260927 03:28:05.827222 2651 raft_consensus.cc:374] T a5083f810bf94b3e9824825f4abb7a31 P 5d9947006e7c4ae89d95618bb8ed6a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:05.827273 2651 raft_consensus.cc:729] T a5083f810bf94b3e9824825f4abb7a31 P 5d9947006e7c4ae89d95618bb8ed6a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d9947006e7c4ae89d95618bb8ed6a48, State: Initialized, Role: FOLLOWER
I20260927 03:28:05.827409 2651 consensus_queue.cc:260] T a5083f810bf94b3e9824825f4abb7a31 P 5d9947006e7c4ae89d95618bb8ed6a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } } peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } }
I20260927 03:28:05.827700 2651 ts_tablet_manager.cc:1428] T a5083f810bf94b3e9824825f4abb7a31 P 5d9947006e7c4ae89d95618bb8ed6a48: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 03:28:05.827885 2346 heartbeater.cc:503] Master 127.0.98.62:42161 was elected leader, sending a full tablet report...
W20260927 03:28:05.869833 2487 tablet.cc:2369] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:28:05.950372 2656 raft_consensus.cc:482] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:28:05.950498 2656 raft_consensus.cc:504] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } } peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } }
I20260927 03:28:05.950826 2656 leader_election.cc:288] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c16c0dcf19b2426e8d54a191b46cf973 (127.0.98.3:42561), 5d9947006e7c4ae89d95618bb8ed6a48 (127.0.98.1:41697)
I20260927 03:28:05.954526 2379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:05.951023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42561 (local address 127.0.98.2:51653)
0927 03:28:05.951078 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:05.951092 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:05.951138 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:05.951382 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:05.951383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:05.951385 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:05.951483 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:05.951485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.952379 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.952379 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:05.953648 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:05.953651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.954241 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.954242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:05.954328 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:05.954331 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:28:05.954376 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 03:28:05.954411 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":21}
I20260927 03:28:05.954531 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:05.951190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51653 (local address 127.0.98.3:42561)
0927 03:28:05.951239 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:05.951241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:05.951251 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:28:05.951280 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:05.951281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:05.951292 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:05.951358 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:05.951549 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.952310 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.953768 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.954152 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.954210 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:05.954244 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:05.954410 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 03:28:05.954460 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:28:05.954658 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:05.951027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41697 (local address 127.0.98.2:42573)
0927 03:28:05.951387 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:05.951399 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:05.951505 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:05.951645 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:05.951646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:05.951647 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:05.951700 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:05.951701 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.952503 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.952504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:05.953648 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:05.953651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.954290 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.954291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:05.954381 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:05.954386 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:28:05.954453 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 03:28:05.954492 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":167,"thread_start_us":111,"threads_started":1}
I20260927 03:28:05.954594 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:05.951110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42573 (local address 127.0.98.1:41697)
0927 03:28:05.951159 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:05.951161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:05.951514 (+ 353us) server_negotiation.cc:407] Connection header received
0927 03:28:05.951565 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:05.951566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:05.951577 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:05.951630 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:05.951763 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.952436 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.953778 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:05.954156 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:05.954248 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:05.954258 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:05.954472 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 03:28:05.954515 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 03:28:05.954959 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5083f810bf94b3e9824825f4abb7a31" candidate_uuid: "74c74bf308644ca1a7c3d6715a1d504f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c16c0dcf19b2426e8d54a191b46cf973" is_pre_election: true
I20260927 03:28:05.955049 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5083f810bf94b3e9824825f4abb7a31" candidate_uuid: "74c74bf308644ca1a7c3d6715a1d504f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" is_pre_election: true
I20260927 03:28:05.955153 2580 raft_consensus.cc:2454] T a5083f810bf94b3e9824825f4abb7a31 P c16c0dcf19b2426e8d54a191b46cf973 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74c74bf308644ca1a7c3d6715a1d504f in term 0.
I20260927 03:28:05.955224 2300 raft_consensus.cc:2454] T a5083f810bf94b3e9824825f4abb7a31 P 5d9947006e7c4ae89d95618bb8ed6a48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74c74bf308644ca1a7c3d6715a1d504f in term 0.
I20260927 03:28:05.955404 2374 leader_election.cc:302] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74c74bf308644ca1a7c3d6715a1d504f, c16c0dcf19b2426e8d54a191b46cf973; no voters:
I20260927 03:28:05.955608 2656 raft_consensus.cc:2781] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:28:05.955673 2656 raft_consensus.cc:482] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:28:05.955693 2656 raft_consensus.cc:3037] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:05.956705 2656 raft_consensus.cc:504] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } } peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } }
I20260927 03:28:05.956868 2656 leader_election.cc:288] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f [CANDIDATE]: Term 1 election: Requested vote from peers c16c0dcf19b2426e8d54a191b46cf973 (127.0.98.3:42561), 5d9947006e7c4ae89d95618bb8ed6a48 (127.0.98.1:41697)
I20260927 03:28:05.957058 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5083f810bf94b3e9824825f4abb7a31" candidate_uuid: "74c74bf308644ca1a7c3d6715a1d504f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d9947006e7c4ae89d95618bb8ed6a48"
I20260927 03:28:05.957068 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5083f810bf94b3e9824825f4abb7a31" candidate_uuid: "74c74bf308644ca1a7c3d6715a1d504f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c16c0dcf19b2426e8d54a191b46cf973"
I20260927 03:28:05.957144 2300 raft_consensus.cc:3037] T a5083f810bf94b3e9824825f4abb7a31 P 5d9947006e7c4ae89d95618bb8ed6a48 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:05.957144 2580 raft_consensus.cc:3037] T a5083f810bf94b3e9824825f4abb7a31 P c16c0dcf19b2426e8d54a191b46cf973 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:05.958041 2580 raft_consensus.cc:2454] T a5083f810bf94b3e9824825f4abb7a31 P c16c0dcf19b2426e8d54a191b46cf973 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74c74bf308644ca1a7c3d6715a1d504f in term 1.
I20260927 03:28:05.958041 2300 raft_consensus.cc:2454] T a5083f810bf94b3e9824825f4abb7a31 P 5d9947006e7c4ae89d95618bb8ed6a48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74c74bf308644ca1a7c3d6715a1d504f in term 1.
I20260927 03:28:05.958236 2371 leader_election.cc:302] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d9947006e7c4ae89d95618bb8ed6a48, 74c74bf308644ca1a7c3d6715a1d504f; no voters:
I20260927 03:28:05.958374 2656 raft_consensus.cc:2781] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:05.958546 2656 raft_consensus.cc:686] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f [term 1 LEADER]: Becoming Leader. State: Replica: 74c74bf308644ca1a7c3d6715a1d504f, State: Running, Role: LEADER
I20260927 03:28:05.958647 2656 consensus_queue.cc:237] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } } peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } }
I20260927 03:28:05.959605 2142 catalog_manager.cc:5495] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f reported cstate change: term changed from 0 to 1, leader changed from <none> to 74c74bf308644ca1a7c3d6715a1d504f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "74c74bf308644ca1a7c3d6715a1d504f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } health_report { overall_health: UNKNOWN } } }
W20260927 03:28:05.963927 2347 tablet.cc:2369] T a5083f810bf94b3e9824825f4abb7a31 P 5d9947006e7c4ae89d95618bb8ed6a48: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:28:06.021667 2627 tablet.cc:2369] T a5083f810bf94b3e9824825f4abb7a31 P c16c0dcf19b2426e8d54a191b46cf973: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 03:28:06 dist-test-slave-j0hf krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:06.035364 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:06.024129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42161 (local address 127.0.0.1:40172)
0927 03:28:06.024292 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:06.024308 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:06.024369 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:06.024532 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:06.024533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:06.024837 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:06.024885 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:06.024887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:06.025502 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:06.025503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:06.026025 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:06.026026 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:06.026325 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:06.026325 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:06.026368 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:06.031364 (+ 4996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:06.031369 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:06.032901 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:06.034395 (+ 1494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:06.034403 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:06.034405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:06.034604 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:06.034814 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:06.034815 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:06.034815 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:06.034870 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:06.035090 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:06.035096 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:06.035139 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 03:28:06.035190 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 03:28:06.035285 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":100,"threads_started":1}
I20260927 03:28:06.035610 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:06.024188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40172 (local address 127.0.98.62:42161)
0927 03:28:06.024260 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:06.024262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:06.024368 (+ 106us) server_negotiation.cc:407] Connection header received
0927 03:28:06.024414 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:06.024414 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:06.024423 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:06.024487 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:06.024959 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:06.025428 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:06.026091 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:06.026260 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:06.026308 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:06.033013 (+ 6705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:06.033015 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:06.033016 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:06.033023 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:06.034318 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:06.034682 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:06.034683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:06.034684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:06.034752 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:06.034925 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:06.034926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:06.034926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:06.035038 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:06.035090 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:06.035220 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 03:28:06.035512 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":164,"server-negotiator.queue_time_us":39}
I20260927 03:28:06.037550 2142 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40172:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 03:28:06.037784 2142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:28:06.039278 2420 tablet_service.cc:1467] Processing CreateTablet for tablet c2485d399096486d92307b6376e6d66f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=74af02104d8546c5adc57766305ae1b6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:28:06.039294 2550 tablet_service.cc:1467] Processing CreateTablet for tablet c2485d399096486d92307b6376e6d66f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=74af02104d8546c5adc57766305ae1b6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:28:06.039318 2280 tablet_service.cc:1467] Processing CreateTablet for tablet c2485d399096486d92307b6376e6d66f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=74af02104d8546c5adc57766305ae1b6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:28:06.039495 2420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2485d399096486d92307b6376e6d66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:06.039511 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2485d399096486d92307b6376e6d66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:06.039564 2550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2485d399096486d92307b6376e6d66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:06.040679 2651 tablet_bootstrap.cc:492] T c2485d399096486d92307b6376e6d66f P 5d9947006e7c4ae89d95618bb8ed6a48: Bootstrap starting.
I20260927 03:28:06.041054 2650 tablet_bootstrap.cc:492] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f: Bootstrap starting.
I20260927 03:28:06.041100 2651 tablet_bootstrap.cc:654] T c2485d399096486d92307b6376e6d66f P 5d9947006e7c4ae89d95618bb8ed6a48: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:06.041484 2650 tablet_bootstrap.cc:654] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:06.041782 2651 tablet_bootstrap.cc:492] T c2485d399096486d92307b6376e6d66f P 5d9947006e7c4ae89d95618bb8ed6a48: No bootstrap required, opened a new log
I20260927 03:28:06.041827 2651 ts_tablet_manager.cc:1397] T c2485d399096486d92307b6376e6d66f P 5d9947006e7c4ae89d95618bb8ed6a48: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 03:28:06.041849 2649 tablet_bootstrap.cc:492] T c2485d399096486d92307b6376e6d66f P c16c0dcf19b2426e8d54a191b46cf973: Bootstrap starting.
I20260927 03:28:06.041973 2651 raft_consensus.cc:348] T c2485d399096486d92307b6376e6d66f P 5d9947006e7c4ae89d95618bb8ed6a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:06.042062 2651 raft_consensus.cc:374] T c2485d399096486d92307b6376e6d66f P 5d9947006e7c4ae89d95618bb8ed6a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:06.042088 2651 raft_consensus.cc:729] T c2485d399096486d92307b6376e6d66f P 5d9947006e7c4ae89d95618bb8ed6a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d9947006e7c4ae89d95618bb8ed6a48, State: Initialized, Role: FOLLOWER
I20260927 03:28:06.042136 2651 consensus_queue.cc:260] T c2485d399096486d92307b6376e6d66f P 5d9947006e7c4ae89d95618bb8ed6a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:06.042256 2651 ts_tablet_manager.cc:1428] T c2485d399096486d92307b6376e6d66f P 5d9947006e7c4ae89d95618bb8ed6a48: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 03:28:06.042291 2650 tablet_bootstrap.cc:492] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f: No bootstrap required, opened a new log
I20260927 03:28:06.042336 2650 ts_tablet_manager.cc:1397] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 03:28:06.042326 2657 tablet_replica.cc:439] T c2485d399096486d92307b6376e6d66f P 5d9947006e7c4ae89d95618bb8ed6a48: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } } }
I20260927 03:28:06.042404 2649 tablet_bootstrap.cc:654] T c2485d399096486d92307b6376e6d66f P c16c0dcf19b2426e8d54a191b46cf973: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:06.042435 2657 tablet_replica.cc:442] T c2485d399096486d92307b6376e6d66f P 5d9947006e7c4ae89d95618bb8ed6a48: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:28:06.042505 2650 raft_consensus.cc:348] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:06.042582 2650 raft_consensus.cc:374] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:06.042601 2650 raft_consensus.cc:729] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c74bf308644ca1a7c3d6715a1d504f, State: Initialized, Role: FOLLOWER
I20260927 03:28:06.042647 2650 consensus_queue.cc:260] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:06.042737 2650 ts_tablet_manager.cc:1428] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:06.042788 2656 tablet_replica.cc:439] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } } }
I20260927 03:28:06.042889 2656 tablet_replica.cc:442] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:28:06.043533 2649 tablet_bootstrap.cc:492] T c2485d399096486d92307b6376e6d66f P c16c0dcf19b2426e8d54a191b46cf973: No bootstrap required, opened a new log
I20260927 03:28:06.043598 2649 ts_tablet_manager.cc:1397] T c2485d399096486d92307b6376e6d66f P c16c0dcf19b2426e8d54a191b46cf973: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:28:06.043743 2649 raft_consensus.cc:348] T c2485d399096486d92307b6376e6d66f P c16c0dcf19b2426e8d54a191b46cf973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:06.043815 2649 raft_consensus.cc:374] T c2485d399096486d92307b6376e6d66f P c16c0dcf19b2426e8d54a191b46cf973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:06.043838 2649 raft_consensus.cc:729] T c2485d399096486d92307b6376e6d66f P c16c0dcf19b2426e8d54a191b46cf973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c16c0dcf19b2426e8d54a191b46cf973, State: Initialized, Role: FOLLOWER
I20260927 03:28:06.043895 2649 consensus_queue.cc:260] T c2485d399096486d92307b6376e6d66f P c16c0dcf19b2426e8d54a191b46cf973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:06.043993 2649 ts_tablet_manager.cc:1428] T c2485d399096486d92307b6376e6d66f P c16c0dcf19b2426e8d54a191b46cf973: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:06.044072 2655 tablet_replica.cc:439] T c2485d399096486d92307b6376e6d66f P c16c0dcf19b2426e8d54a191b46cf973: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } } }
I20260927 03:28:06.044169 2655 tablet_replica.cc:442] T c2485d399096486d92307b6376e6d66f P c16c0dcf19b2426e8d54a191b46cf973: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:28:06.134124 2656 raft_consensus.cc:482] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:28:06.134239 2656 raft_consensus.cc:504] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:06.134428 2656 leader_election.cc:288] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d9947006e7c4ae89d95618bb8ed6a48 (127.0.98.1:41697), c16c0dcf19b2426e8d54a191b46cf973 (127.0.98.3:42561)
I20260927 03:28:06.134665 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2485d399096486d92307b6376e6d66f" candidate_uuid: "74c74bf308644ca1a7c3d6715a1d504f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c16c0dcf19b2426e8d54a191b46cf973" is_pre_election: true
I20260927 03:28:06.134665 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2485d399096486d92307b6376e6d66f" candidate_uuid: "74c74bf308644ca1a7c3d6715a1d504f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" is_pre_election: true
I20260927 03:28:06.134766 2580 raft_consensus.cc:2454] T c2485d399096486d92307b6376e6d66f P c16c0dcf19b2426e8d54a191b46cf973 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74c74bf308644ca1a7c3d6715a1d504f in term 0.
I20260927 03:28:06.134766 2300 raft_consensus.cc:2454] T c2485d399096486d92307b6376e6d66f P 5d9947006e7c4ae89d95618bb8ed6a48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74c74bf308644ca1a7c3d6715a1d504f in term 0.
I20260927 03:28:06.134928 2371 leader_election.cc:302] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d9947006e7c4ae89d95618bb8ed6a48, 74c74bf308644ca1a7c3d6715a1d504f; no voters:
I20260927 03:28:06.135059 2656 raft_consensus.cc:2781] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:28:06.135104 2656 raft_consensus.cc:482] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:28:06.135138 2656 raft_consensus.cc:3037] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:06.135872 2656 raft_consensus.cc:504] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:06.136026 2656 leader_election.cc:288] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f [CANDIDATE]: Term 1 election: Requested vote from peers 5d9947006e7c4ae89d95618bb8ed6a48 (127.0.98.1:41697), c16c0dcf19b2426e8d54a191b46cf973 (127.0.98.3:42561)
I20260927 03:28:06.136170 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2485d399096486d92307b6376e6d66f" candidate_uuid: "74c74bf308644ca1a7c3d6715a1d504f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d9947006e7c4ae89d95618bb8ed6a48"
I20260927 03:28:06.136270 2300 raft_consensus.cc:3037] T c2485d399096486d92307b6376e6d66f P 5d9947006e7c4ae89d95618bb8ed6a48 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:06.136240 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2485d399096486d92307b6376e6d66f" candidate_uuid: "74c74bf308644ca1a7c3d6715a1d504f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c16c0dcf19b2426e8d54a191b46cf973"
I20260927 03:28:06.136335 2580 raft_consensus.cc:3037] T c2485d399096486d92307b6376e6d66f P c16c0dcf19b2426e8d54a191b46cf973 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:06.137027 2300 raft_consensus.cc:2454] T c2485d399096486d92307b6376e6d66f P 5d9947006e7c4ae89d95618bb8ed6a48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74c74bf308644ca1a7c3d6715a1d504f in term 1.
I20260927 03:28:06.137094 2580 raft_consensus.cc:2454] T c2485d399096486d92307b6376e6d66f P c16c0dcf19b2426e8d54a191b46cf973 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74c74bf308644ca1a7c3d6715a1d504f in term 1.
I20260927 03:28:06.137291 2371 leader_election.cc:302] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5d9947006e7c4ae89d95618bb8ed6a48, 74c74bf308644ca1a7c3d6715a1d504f, c16c0dcf19b2426e8d54a191b46cf973; no voters:
I20260927 03:28:06.137395 2656 raft_consensus.cc:2781] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:06.137461 2656 raft_consensus.cc:686] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f [term 1 LEADER]: Becoming Leader. State: Replica: 74c74bf308644ca1a7c3d6715a1d504f, State: Running, Role: LEADER
I20260927 03:28:06.137517 2656 consensus_queue.cc:237] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:06.137835 2659 tablet_replica.cc:439] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f: TxnStatusTablet state changed. Reason: New leader 74c74bf308644ca1a7c3d6715a1d504f. Latest consensus state: current_term: 1 leader_uuid: "74c74bf308644ca1a7c3d6715a1d504f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } } }
I20260927 03:28:06.137931 2659 tablet_replica.cc:442] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:28:06.138085 2671 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 03:28:06.138255 2671 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 03:28:06.138293 2143 catalog_manager.cc:5495] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f reported cstate change: term changed from 0 to 1, leader changed from <none> to 74c74bf308644ca1a7c3d6715a1d504f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "74c74bf308644ca1a7c3d6715a1d504f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } health_report { overall_health: UNKNOWN } } }
I20260927 03:28:06.161645 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:06.158759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58860 (local address 127.0.98.2:42071)
0927 03:28:06.158823 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:06.158825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:06.158833 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:28:06.158940 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:06.158940 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:06.158958 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:06.159018 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:28:06.159159 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:06.159703 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:06.160868 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:06.161056 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:06.161109 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:06.161317 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:28:06.161411 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:28:06.161472 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:06.161550 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 03:28:06.161567 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 03:28:06.161686 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:06.158697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42071 (local address 127.0.0.1:58860)
0927 03:28:06.158779 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:06.158794 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:06.158848 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:06.159041 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:06.159042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:06.159043 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:28:06.159097 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:06.159100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:06.159792 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:06.159794 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:06.160775 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:06.160779 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:06.161135 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:06.161136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:06.161207 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:06.161226 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:28:06.161483 (+ 257us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:28:06.161490 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 03:28:06.161545 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 03:28:06.161596 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 03:28:06.163602 2580 raft_consensus.cc:1261] T c2485d399096486d92307b6376e6d66f P c16c0dcf19b2426e8d54a191b46cf973 [term 1 FOLLOWER]: Refusing update from remote peer 74c74bf308644ca1a7c3d6715a1d504f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:06.163605 2300 raft_consensus.cc:1261] T c2485d399096486d92307b6376e6d66f P 5d9947006e7c4ae89d95618bb8ed6a48 [term 1 FOLLOWER]: Refusing update from remote peer 74c74bf308644ca1a7c3d6715a1d504f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:06.163971 2659 consensus_queue.cc:1035] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:06.164095 2656 consensus_queue.cc:1035] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:06.165374 2655 tablet_replica.cc:439] T c2485d399096486d92307b6376e6d66f P c16c0dcf19b2426e8d54a191b46cf973: TxnStatusTablet state changed. Reason: New leader 74c74bf308644ca1a7c3d6715a1d504f. Latest consensus state: current_term: 1 leader_uuid: "74c74bf308644ca1a7c3d6715a1d504f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } } }
I20260927 03:28:06.165498 2655 tablet_replica.cc:442] T c2485d399096486d92307b6376e6d66f P c16c0dcf19b2426e8d54a191b46cf973: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:28:06.165553 2657 tablet_replica.cc:439] T c2485d399096486d92307b6376e6d66f P 5d9947006e7c4ae89d95618bb8ed6a48: TxnStatusTablet state changed. Reason: New leader 74c74bf308644ca1a7c3d6715a1d504f. Latest consensus state: current_term: 1 leader_uuid: "74c74bf308644ca1a7c3d6715a1d504f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } } }
I20260927 03:28:06.165618 2657 tablet_replica.cc:442] T c2485d399096486d92307b6376e6d66f P 5d9947006e7c4ae89d95618bb8ed6a48: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:28:06.167125 2659 tablet_replica.cc:439] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "74c74bf308644ca1a7c3d6715a1d504f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } } }
I20260927 03:28:06.167217 2659 tablet_replica.cc:442] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:28:06.167321 2659 tablet_replica.cc:439] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "74c74bf308644ca1a7c3d6715a1d504f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } } }
I20260927 03:28:06.167366 2659 tablet_replica.cc:442] T c2485d399096486d92307b6376e6d66f P 74c74bf308644ca1a7c3d6715a1d504f: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:28:06.167352 2657 tablet_replica.cc:439] T c2485d399096486d92307b6376e6d66f P 5d9947006e7c4ae89d95618bb8ed6a48: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "74c74bf308644ca1a7c3d6715a1d504f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } } }
I20260927 03:28:06.167456 2657 tablet_replica.cc:442] T c2485d399096486d92307b6376e6d66f P 5d9947006e7c4ae89d95618bb8ed6a48: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:28:06.167939 2675 mvcc.cc:205] Tried to move back new op lower bound from 7333804794523557888 to 7333804794419814400. Current Snapshot: MvccSnapshot[applied={T|T < 7333804794523557888}]
I20260927 03:28:06.168459 2676 mvcc.cc:205] Tried to move back new op lower bound from 7333804794523557888 to 7333804794419814400. Current Snapshot: MvccSnapshot[applied={T|T < 7333804794523557888}]
I20260927 03:28:06.168551 2655 tablet_replica.cc:439] T c2485d399096486d92307b6376e6d66f P c16c0dcf19b2426e8d54a191b46cf973: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "74c74bf308644ca1a7c3d6715a1d504f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } } }
I20260927 03:28:06.168615 2655 tablet_replica.cc:442] T c2485d399096486d92307b6376e6d66f P c16c0dcf19b2426e8d54a191b46cf973: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:28:06.169100 2674 mvcc.cc:205] Tried to move back new op lower bound from 7333804794523557888 to 7333804794419814400. Current Snapshot: MvccSnapshot[applied={T|T < 7333804794523557888 or (T in {7333804794523557888})}]
I20260927 03:28:06.172163 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:06.169800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58862 (local address 127.0.98.2:42071)
0927 03:28:06.169863 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:06.169865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:06.169874 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:28:06.169952 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:06.169953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:06.169964 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:06.170007 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:28:06.170153 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:06.170647 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:06.171413 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:06.171645 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:06.171721 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:06.171887 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:28:06.171921 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:28:06.171983 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:06.172062 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 03:28:06.172082 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:28:06.177700 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:06.174761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37469 (local address 127.0.98.2:42071)
0927 03:28:06.174795 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:06.174796 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:06.174938 (+ 142us) server_negotiation.cc:407] Connection header received
0927 03:28:06.175002 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:06.175003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:06.175011 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:06.175058 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:28:06.175212 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:06.175719 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:06.176856 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:06.177030 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:06.177142 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:06.177428 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:28:06.177462 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:28:06.177522 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:06.177599 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 03:28:06.177619 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":14}
I20260927 03:28:06.178016 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:06.174696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42071 (local address 127.0.98.2:37469)
0927 03:28:06.174878 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:06.174889 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:06.174946 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:06.175081 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:06.175082 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:06.175083 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:28:06.175150 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:06.175152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:06.175787 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:06.175788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:06.176756 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:06.176759 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:06.177157 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:06.177158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:06.177236 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:06.177251 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:28:06.177539 (+ 288us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:28:06.177541 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:28:06.177597 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 03:28:06.177644 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":1,"thread_start_us":82,"threads_started":1}
I20260927 03:28:06.180338 2580 raft_consensus.cc:1261] T a5083f810bf94b3e9824825f4abb7a31 P c16c0dcf19b2426e8d54a191b46cf973 [term 1 FOLLOWER]: Refusing update from remote peer 74c74bf308644ca1a7c3d6715a1d504f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:06.180495 2300 raft_consensus.cc:1261] T a5083f810bf94b3e9824825f4abb7a31 P 5d9947006e7c4ae89d95618bb8ed6a48 [term 1 FOLLOWER]: Refusing update from remote peer 74c74bf308644ca1a7c3d6715a1d504f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:06.180581 2656 consensus_queue.cc:1035] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:06.180680 2659 consensus_queue.cc:1035] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:06.194876 2143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:28:06.194995 2143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:28:06.195818 2143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80fc0bdfbbfc4e259a2ac9cb3841e2d5: Sending DeleteTablet for 3 replicas of tablet a5083f810bf94b3e9824825f4abb7a31
I20260927 03:28:06.196149 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet a5083f810bf94b3e9824825f4abb7a31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42052
I20260927 03:28:06.196149 2550 tablet_service.cc:1514] Processing DeleteTablet for tablet a5083f810bf94b3e9824825f4abb7a31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39962
I20260927 03:28:06.196326 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:28:06.196482 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:28:06.196550 2420 tablet_service.cc:1514] Processing DeleteTablet for tablet a5083f810bf94b3e9824825f4abb7a31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58852
I20260927 03:28:06.196772 2692 tablet_replica.cc:330] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f: stopping tablet replica
I20260927 03:28:06.196882 2692 raft_consensus.cc:2229] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:28:06.196995 2692 raft_consensus.cc:2258] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:06.197425 2690 tablet_replica.cc:330] T a5083f810bf94b3e9824825f4abb7a31 P 5d9947006e7c4ae89d95618bb8ed6a48: stopping tablet replica
I20260927 03:28:06.197584 2550 tablet_service.cc:1467] Processing CreateTablet for tablet 871eabae933143fab0d0b6ae9014dbbc (DEFAULT_TABLE table=test-table [id=6c5bdd94635641c78a26080c560509d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:06.197708 2550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 871eabae933143fab0d0b6ae9014dbbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:06.197850 2692 ts_tablet_manager.cc:1905] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:06.199149 2649 tablet_bootstrap.cc:492] T 871eabae933143fab0d0b6ae9014dbbc P c16c0dcf19b2426e8d54a191b46cf973: Bootstrap starting.
I20260927 03:28:06.199497 2420 tablet_service.cc:1467] Processing CreateTablet for tablet 871eabae933143fab0d0b6ae9014dbbc (DEFAULT_TABLE table=test-table [id=6c5bdd94635641c78a26080c560509d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:06.199608 2649 tablet_bootstrap.cc:654] T 871eabae933143fab0d0b6ae9014dbbc P c16c0dcf19b2426e8d54a191b46cf973: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:06.199669 2691 tablet_replica.cc:330] T a5083f810bf94b3e9824825f4abb7a31 P c16c0dcf19b2426e8d54a191b46cf973: stopping tablet replica
I20260927 03:28:06.199797 2691 raft_consensus.cc:2229] T a5083f810bf94b3e9824825f4abb7a31 P c16c0dcf19b2426e8d54a191b46cf973 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:28:06.199882 2691 raft_consensus.cc:2258] T a5083f810bf94b3e9824825f4abb7a31 P c16c0dcf19b2426e8d54a191b46cf973 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:06.197506 2280 tablet_service.cc:1467] Processing CreateTablet for tablet 871eabae933143fab0d0b6ae9014dbbc (DEFAULT_TABLE table=test-table [id=6c5bdd94635641c78a26080c560509d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:06.200073 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 871eabae933143fab0d0b6ae9014dbbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:06.201083 2651 tablet_bootstrap.cc:492] T 871eabae933143fab0d0b6ae9014dbbc P 5d9947006e7c4ae89d95618bb8ed6a48: Bootstrap starting.
I20260927 03:28:06.201236 2690 raft_consensus.cc:2229] T a5083f810bf94b3e9824825f4abb7a31 P 5d9947006e7c4ae89d95618bb8ed6a48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:28:06.201333 2690 raft_consensus.cc:2258] T a5083f810bf94b3e9824825f4abb7a31 P 5d9947006e7c4ae89d95618bb8ed6a48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:06.201469 2651 tablet_bootstrap.cc:654] T 871eabae933143fab0d0b6ae9014dbbc P 5d9947006e7c4ae89d95618bb8ed6a48: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:06.201623 2691 ts_tablet_manager.cc:1905] T a5083f810bf94b3e9824825f4abb7a31 P c16c0dcf19b2426e8d54a191b46cf973: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:06.202070 2651 tablet_bootstrap.cc:492] T 871eabae933143fab0d0b6ae9014dbbc P 5d9947006e7c4ae89d95618bb8ed6a48: No bootstrap required, opened a new log
I20260927 03:28:06.202121 2651 ts_tablet_manager.cc:1397] T 871eabae933143fab0d0b6ae9014dbbc P 5d9947006e7c4ae89d95618bb8ed6a48: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 03:28:06.202245 2651 raft_consensus.cc:348] T 871eabae933143fab0d0b6ae9014dbbc P 5d9947006e7c4ae89d95618bb8ed6a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:06.202313 2651 raft_consensus.cc:374] T 871eabae933143fab0d0b6ae9014dbbc P 5d9947006e7c4ae89d95618bb8ed6a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:06.202335 2651 raft_consensus.cc:729] T 871eabae933143fab0d0b6ae9014dbbc P 5d9947006e7c4ae89d95618bb8ed6a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d9947006e7c4ae89d95618bb8ed6a48, State: Initialized, Role: FOLLOWER
I20260927 03:28:06.202376 2651 consensus_queue.cc:260] T 871eabae933143fab0d0b6ae9014dbbc P 5d9947006e7c4ae89d95618bb8ed6a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:06.202467 2651 ts_tablet_manager.cc:1428] T 871eabae933143fab0d0b6ae9014dbbc P 5d9947006e7c4ae89d95618bb8ed6a48: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:06.203140 2691 ts_tablet_manager.cc:1918] T a5083f810bf94b3e9824825f4abb7a31 P c16c0dcf19b2426e8d54a191b46cf973: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:28:06.203225 2691 log.cc:1196] T a5083f810bf94b3e9824825f4abb7a31 P c16c0dcf19b2426e8d54a191b46cf973: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-2/wal/wals/a5083f810bf94b3e9824825f4abb7a31
I20260927 03:28:06.203498 2691 ts_tablet_manager.cc:1939] T a5083f810bf94b3e9824825f4abb7a31 P c16c0dcf19b2426e8d54a191b46cf973: Deleting consensus metadata
I20260927 03:28:06.203789 2649 tablet_bootstrap.cc:492] T 871eabae933143fab0d0b6ae9014dbbc P c16c0dcf19b2426e8d54a191b46cf973: No bootstrap required, opened a new log
I20260927 03:28:06.203851 2649 ts_tablet_manager.cc:1397] T 871eabae933143fab0d0b6ae9014dbbc P c16c0dcf19b2426e8d54a191b46cf973: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260927 03:28:06.203909 2130 catalog_manager.cc:4841] TS c16c0dcf19b2426e8d54a191b46cf973 (127.0.98.3:42561): tablet a5083f810bf94b3e9824825f4abb7a31 (table test-table [id=ead9a8b1e77a41bfaacf58f20be90e2b]) successfully deleted
I20260927 03:28:06.203979 2649 raft_consensus.cc:348] T 871eabae933143fab0d0b6ae9014dbbc P c16c0dcf19b2426e8d54a191b46cf973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:06.204046 2649 raft_consensus.cc:374] T 871eabae933143fab0d0b6ae9014dbbc P c16c0dcf19b2426e8d54a191b46cf973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:06.204072 2649 raft_consensus.cc:729] T 871eabae933143fab0d0b6ae9014dbbc P c16c0dcf19b2426e8d54a191b46cf973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c16c0dcf19b2426e8d54a191b46cf973, State: Initialized, Role: FOLLOWER
I20260927 03:28:06.204077 2690 ts_tablet_manager.cc:1905] T a5083f810bf94b3e9824825f4abb7a31 P 5d9947006e7c4ae89d95618bb8ed6a48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:06.204111 2649 consensus_queue.cc:260] T 871eabae933143fab0d0b6ae9014dbbc P c16c0dcf19b2426e8d54a191b46cf973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:06.204192 2649 ts_tablet_manager.cc:1428] T 871eabae933143fab0d0b6ae9014dbbc P c16c0dcf19b2426e8d54a191b46cf973: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 03:28:06.205560 2690 ts_tablet_manager.cc:1918] T a5083f810bf94b3e9824825f4abb7a31 P 5d9947006e7c4ae89d95618bb8ed6a48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:28:06.205542 2692 ts_tablet_manager.cc:1918] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:28:06.205542 2420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 871eabae933143fab0d0b6ae9014dbbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:06.205623 2690 log.cc:1196] T a5083f810bf94b3e9824825f4abb7a31 P 5d9947006e7c4ae89d95618bb8ed6a48: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-0/wal/wals/a5083f810bf94b3e9824825f4abb7a31
I20260927 03:28:06.205638 2692 log.cc:1196] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-1/wal/wals/a5083f810bf94b3e9824825f4abb7a31
I20260927 03:28:06.205865 2690 ts_tablet_manager.cc:1939] T a5083f810bf94b3e9824825f4abb7a31 P 5d9947006e7c4ae89d95618bb8ed6a48: Deleting consensus metadata
I20260927 03:28:06.205888 2692 ts_tablet_manager.cc:1939] T a5083f810bf94b3e9824825f4abb7a31 P 74c74bf308644ca1a7c3d6715a1d504f: Deleting consensus metadata
I20260927 03:28:06.206241 2127 catalog_manager.cc:4841] TS 5d9947006e7c4ae89d95618bb8ed6a48 (127.0.98.1:41697): tablet a5083f810bf94b3e9824825f4abb7a31 (table test-table [id=ead9a8b1e77a41bfaacf58f20be90e2b]) successfully deleted
I20260927 03:28:06.206732 2650 tablet_bootstrap.cc:492] T 871eabae933143fab0d0b6ae9014dbbc P 74c74bf308644ca1a7c3d6715a1d504f: Bootstrap starting.
I20260927 03:28:06.206836 2130 catalog_manager.cc:4841] TS 74c74bf308644ca1a7c3d6715a1d504f (127.0.98.2:42071): tablet a5083f810bf94b3e9824825f4abb7a31 (table test-table [id=ead9a8b1e77a41bfaacf58f20be90e2b]) successfully deleted
I20260927 03:28:06.207095 2650 tablet_bootstrap.cc:654] T 871eabae933143fab0d0b6ae9014dbbc P 74c74bf308644ca1a7c3d6715a1d504f: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:06.207733 2650 tablet_bootstrap.cc:492] T 871eabae933143fab0d0b6ae9014dbbc P 74c74bf308644ca1a7c3d6715a1d504f: No bootstrap required, opened a new log
I20260927 03:28:06.207787 2650 ts_tablet_manager.cc:1397] T 871eabae933143fab0d0b6ae9014dbbc P 74c74bf308644ca1a7c3d6715a1d504f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 03:28:06.207938 2650 raft_consensus.cc:348] T 871eabae933143fab0d0b6ae9014dbbc P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:06.208029 2650 raft_consensus.cc:374] T 871eabae933143fab0d0b6ae9014dbbc P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:06.208056 2650 raft_consensus.cc:729] T 871eabae933143fab0d0b6ae9014dbbc P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c74bf308644ca1a7c3d6715a1d504f, State: Initialized, Role: FOLLOWER
I20260927 03:28:06.208101 2650 consensus_queue.cc:260] T 871eabae933143fab0d0b6ae9014dbbc P 74c74bf308644ca1a7c3d6715a1d504f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:06.208195 2650 ts_tablet_manager.cc:1428] T 871eabae933143fab0d0b6ae9014dbbc P 74c74bf308644ca1a7c3d6715a1d504f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 03:28:06.649976 2657 raft_consensus.cc:482] T 871eabae933143fab0d0b6ae9014dbbc P 5d9947006e7c4ae89d95618bb8ed6a48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:28:06.650090 2657 raft_consensus.cc:504] T 871eabae933143fab0d0b6ae9014dbbc P 5d9947006e7c4ae89d95618bb8ed6a48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:06.650477 2657 leader_election.cc:288] T 871eabae933143fab0d0b6ae9014dbbc P 5d9947006e7c4ae89d95618bb8ed6a48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74c74bf308644ca1a7c3d6715a1d504f (127.0.98.2:42071), c16c0dcf19b2426e8d54a191b46cf973 (127.0.98.3:42561)
I20260927 03:28:06.653650 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:06.650768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41959 (local address 127.0.98.2:42071)
0927 03:28:06.650844 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:06.650846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:06.650946 (+ 100us) server_negotiation.cc:407] Connection header received
0927 03:28:06.651022 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:06.651023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:06.651044 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:06.651112 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:06.651250 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:06.651787 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:06.652884 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:06.653310 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:06.653384 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:06.653398 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:06.653534 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 03:28:06.653572 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 03:28:06.653682 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:06.650974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38087 (local address 127.0.98.3:42561)
0927 03:28:06.651161 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:06.651162 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:06.651168 (+ 6us) server_negotiation.cc:407] Connection header received
0927 03:28:06.651189 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:06.651190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:06.651197 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:06.651283 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:06.651458 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:06.651931 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:06.652894 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:06.653333 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:06.653410 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:06.653448 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:06.653568 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 03:28:06.653606 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":52,"threads_started":1}
I20260927 03:28:06.653702 2696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:06.650678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42071 (local address 127.0.98.1:41959)
0927 03:28:06.650841 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:06.650854 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:06.650958 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:06.651114 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:06.651115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:06.651117 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:06.651174 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:06.651176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:06.651870 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:06.651872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:06.652789 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:06.652792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:06.653397 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:06.653398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:06.653475 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:06.653479 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:28:06.653528 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 03:28:06.653570 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":4,"thread_start_us":87,"threads_started":1}
I20260927 03:28:06.653717 2697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:06.650895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42561 (local address 127.0.98.1:38087)
0927 03:28:06.651005 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:06.651015 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:06.651069 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:06.651288 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:06.651288 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:06.651290 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:06.651345 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:06.651346 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:06.652029 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:06.652030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:06.652807 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:06.652809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:06.653421 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:06.653422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:06.653494 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:06.653497 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:28:06.653563 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 03:28:06.653602 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":1,"thread_start_us":40,"threads_started":1}
I20260927 03:28:06.654030 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "871eabae933143fab0d0b6ae9014dbbc" candidate_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c74bf308644ca1a7c3d6715a1d504f" is_pre_election: true
I20260927 03:28:06.654178 2440 raft_consensus.cc:2454] T 871eabae933143fab0d0b6ae9014dbbc P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d9947006e7c4ae89d95618bb8ed6a48 in term 0.
I20260927 03:28:06.654196 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "871eabae933143fab0d0b6ae9014dbbc" candidate_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c16c0dcf19b2426e8d54a191b46cf973" is_pre_election: true
I20260927 03:28:06.654277 2580 raft_consensus.cc:2454] T 871eabae933143fab0d0b6ae9014dbbc P c16c0dcf19b2426e8d54a191b46cf973 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d9947006e7c4ae89d95618bb8ed6a48 in term 0.
I20260927 03:28:06.654407 2234 leader_election.cc:302] T 871eabae933143fab0d0b6ae9014dbbc P 5d9947006e7c4ae89d95618bb8ed6a48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d9947006e7c4ae89d95618bb8ed6a48, 74c74bf308644ca1a7c3d6715a1d504f; no voters:
I20260927 03:28:06.654567 2657 raft_consensus.cc:2781] T 871eabae933143fab0d0b6ae9014dbbc P 5d9947006e7c4ae89d95618bb8ed6a48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:28:06.654623 2657 raft_consensus.cc:482] T 871eabae933143fab0d0b6ae9014dbbc P 5d9947006e7c4ae89d95618bb8ed6a48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:28:06.654639 2657 raft_consensus.cc:3037] T 871eabae933143fab0d0b6ae9014dbbc P 5d9947006e7c4ae89d95618bb8ed6a48 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:06.655328 2657 raft_consensus.cc:504] T 871eabae933143fab0d0b6ae9014dbbc P 5d9947006e7c4ae89d95618bb8ed6a48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:06.655496 2657 leader_election.cc:288] T 871eabae933143fab0d0b6ae9014dbbc P 5d9947006e7c4ae89d95618bb8ed6a48 [CANDIDATE]: Term 1 election: Requested vote from peers 74c74bf308644ca1a7c3d6715a1d504f (127.0.98.2:42071), c16c0dcf19b2426e8d54a191b46cf973 (127.0.98.3:42561)
I20260927 03:28:06.655676 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "871eabae933143fab0d0b6ae9014dbbc" candidate_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c74bf308644ca1a7c3d6715a1d504f"
I20260927 03:28:06.655686 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "871eabae933143fab0d0b6ae9014dbbc" candidate_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c16c0dcf19b2426e8d54a191b46cf973"
I20260927 03:28:06.655756 2580 raft_consensus.cc:3037] T 871eabae933143fab0d0b6ae9014dbbc P c16c0dcf19b2426e8d54a191b46cf973 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:06.655756 2440 raft_consensus.cc:3037] T 871eabae933143fab0d0b6ae9014dbbc P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:06.656498 2580 raft_consensus.cc:2454] T 871eabae933143fab0d0b6ae9014dbbc P c16c0dcf19b2426e8d54a191b46cf973 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d9947006e7c4ae89d95618bb8ed6a48 in term 1.
I20260927 03:28:06.656574 2440 raft_consensus.cc:2454] T 871eabae933143fab0d0b6ae9014dbbc P 74c74bf308644ca1a7c3d6715a1d504f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d9947006e7c4ae89d95618bb8ed6a48 in term 1.
I20260927 03:28:06.656698 2234 leader_election.cc:302] T 871eabae933143fab0d0b6ae9014dbbc P 5d9947006e7c4ae89d95618bb8ed6a48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d9947006e7c4ae89d95618bb8ed6a48, 74c74bf308644ca1a7c3d6715a1d504f; no voters:
I20260927 03:28:06.656798 2657 raft_consensus.cc:2781] T 871eabae933143fab0d0b6ae9014dbbc P 5d9947006e7c4ae89d95618bb8ed6a48 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:06.656895 2657 raft_consensus.cc:686] T 871eabae933143fab0d0b6ae9014dbbc P 5d9947006e7c4ae89d95618bb8ed6a48 [term 1 LEADER]: Becoming Leader. State: Replica: 5d9947006e7c4ae89d95618bb8ed6a48, State: Running, Role: LEADER
I20260927 03:28:06.656975 2657 consensus_queue.cc:237] T 871eabae933143fab0d0b6ae9014dbbc P 5d9947006e7c4ae89d95618bb8ed6a48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:06.657702 2143 catalog_manager.cc:5495] T 871eabae933143fab0d0b6ae9014dbbc P 5d9947006e7c4ae89d95618bb8ed6a48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d9947006e7c4ae89d95618bb8ed6a48 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } health_report { overall_health: UNKNOWN } } }
I20260927 03:28:06.755067 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:06.752586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52682 (local address 127.0.98.1:41697)
0927 03:28:06.752682 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:06.752684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:06.752747 (+ 63us) server_negotiation.cc:407] Connection header received
0927 03:28:06.752872 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:06.752876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:06.752896 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:06.752971 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:28:06.753069 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:06.753572 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:06.754345 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:06.754545 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:06.754597 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:06.754706 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:28:06.754812 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:28:06.754863 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:06.754932 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 03:28:06.754947 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":48,"threads_started":1}
I20260927 03:28:06.759441 2714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:06.757336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42071 (local address 127.0.98.1:36503)
0927 03:28:06.757512 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:06.757522 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:06.757570 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:06.757680 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:06.757681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:06.757682 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:28:06.757726 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:06.757727 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:06.758251 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:06.758252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:06.758788 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:06.758789 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:06.759097 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:06.759098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:06.759141 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:06.759153 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:28:06.759296 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:28:06.759298 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:28:06.759331 (+ 33us) client_negotiation.cc:240] Negotiation successful
0927 03:28:06.759365 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":102,"threads_started":1}
I20260927 03:28:06.759413 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:06.757432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36503 (local address 127.0.98.2:42071)
0927 03:28:06.757478 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:06.757480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:06.757562 (+ 82us) server_negotiation.cc:407] Connection header received
0927 03:28:06.757619 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:06.757619 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:06.757627 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:06.757664 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:28:06.757765 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:06.758196 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:06.758858 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:06.759033 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:06.759082 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:06.759221 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:28:06.759245 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:28:06.759279 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:06.759332 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 03:28:06.759346 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 03:28:06.761374 2580 raft_consensus.cc:1261] T 871eabae933143fab0d0b6ae9014dbbc P c16c0dcf19b2426e8d54a191b46cf973 [term 1 FOLLOWER]: Refusing update from remote peer 5d9947006e7c4ae89d95618bb8ed6a48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:06.761443 2440 raft_consensus.cc:1261] T 871eabae933143fab0d0b6ae9014dbbc P 74c74bf308644ca1a7c3d6715a1d504f [term 1 FOLLOWER]: Refusing update from remote peer 5d9947006e7c4ae89d95618bb8ed6a48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:06.761655 2699 consensus_queue.cc:1035] T 871eabae933143fab0d0b6ae9014dbbc P 5d9947006e7c4ae89d95618bb8ed6a48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:06.761790 2657 consensus_queue.cc:1035] T 871eabae933143fab0d0b6ae9014dbbc P 5d9947006e7c4ae89d95618bb8ed6a48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:06.768608 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:06.766248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33345 (local address 127.0.98.1:41697)
0927 03:28:06.766289 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:06.766291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:06.766390 (+ 99us) server_negotiation.cc:407] Connection header received
0927 03:28:06.766463 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:06.766463 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:06.766474 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:06.766532 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:28:06.766612 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:06.767044 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:06.767900 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:06.768157 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:06.768252 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:06.768398 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:28:06.768428 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:28:06.768474 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:06.768528 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 03:28:06.768546 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:28:06.768648 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:06.766167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41697 (local address 127.0.98.2:33345)
0927 03:28:06.766334 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:06.766344 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:06.766402 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:06.766533 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:06.766533 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:06.766534 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:28:06.766569 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:06.766570 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:06.767147 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:06.767147 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:06.767801 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:06.767804 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:06.768252 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:06.768253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:06.768313 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:06.768317 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:28:06.768473 (+ 156us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:28:06.768474 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 03:28:06.768524 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 03:28:06.768566 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":87,"threads_started":1}
W20260927 03:28:06.769033 2704 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 03:28:08.290525 2143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:28:08.290645 2143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:28:08.291651 2143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80fc0bdfbbfc4e259a2ac9cb3841e2d5: Sending DeleteTablet for 3 replicas of tablet 871eabae933143fab0d0b6ae9014dbbc
I20260927 03:28:08.291965 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet 871eabae933143fab0d0b6ae9014dbbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42052
I20260927 03:28:08.291988 2420 tablet_service.cc:1514] Processing DeleteTablet for tablet 871eabae933143fab0d0b6ae9014dbbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58852
I20260927 03:28:08.292038 2550 tablet_service.cc:1514] Processing DeleteTablet for tablet 871eabae933143fab0d0b6ae9014dbbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39962
I20260927 03:28:08.292201 2744 tablet_replica.cc:330] T 871eabae933143fab0d0b6ae9014dbbc P 5d9947006e7c4ae89d95618bb8ed6a48: stopping tablet replica
I20260927 03:28:08.292212 2746 tablet_replica.cc:330] T 871eabae933143fab0d0b6ae9014dbbc P c16c0dcf19b2426e8d54a191b46cf973: stopping tablet replica
I20260927 03:28:08.292256 2746 raft_consensus.cc:2229] T 871eabae933143fab0d0b6ae9014dbbc P c16c0dcf19b2426e8d54a191b46cf973 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:28:08.292274 2744 raft_consensus.cc:2229] T 871eabae933143fab0d0b6ae9014dbbc P 5d9947006e7c4ae89d95618bb8ed6a48 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:28:08.292330 2746 raft_consensus.cc:2258] T 871eabae933143fab0d0b6ae9014dbbc P c16c0dcf19b2426e8d54a191b46cf973 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:08.292382 2744 raft_consensus.cc:2258] T 871eabae933143fab0d0b6ae9014dbbc P 5d9947006e7c4ae89d95618bb8ed6a48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:08.292369 2745 tablet_replica.cc:330] T 871eabae933143fab0d0b6ae9014dbbc P 74c74bf308644ca1a7c3d6715a1d504f: stopping tablet replica
I20260927 03:28:08.292443 2745 raft_consensus.cc:2229] T 871eabae933143fab0d0b6ae9014dbbc P 74c74bf308644ca1a7c3d6715a1d504f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:28:08.292483 2745 raft_consensus.cc:2258] T 871eabae933143fab0d0b6ae9014dbbc P 74c74bf308644ca1a7c3d6715a1d504f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:08.292625 2746 ts_tablet_manager.cc:1905] T 871eabae933143fab0d0b6ae9014dbbc P c16c0dcf19b2426e8d54a191b46cf973: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:08.292712 2745 ts_tablet_manager.cc:1905] T 871eabae933143fab0d0b6ae9014dbbc P 74c74bf308644ca1a7c3d6715a1d504f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:08.292831 2744 ts_tablet_manager.cc:1905] T 871eabae933143fab0d0b6ae9014dbbc P 5d9947006e7c4ae89d95618bb8ed6a48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:08.294590 2746 ts_tablet_manager.cc:1918] T 871eabae933143fab0d0b6ae9014dbbc P c16c0dcf19b2426e8d54a191b46cf973: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 03:28:08.294591 2745 ts_tablet_manager.cc:1918] T 871eabae933143fab0d0b6ae9014dbbc P 74c74bf308644ca1a7c3d6715a1d504f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 03:28:08.294651 2745 log.cc:1196] T 871eabae933143fab0d0b6ae9014dbbc P 74c74bf308644ca1a7c3d6715a1d504f: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-1/wal/wals/871eabae933143fab0d0b6ae9014dbbc
I20260927 03:28:08.294691 2746 log.cc:1196] T 871eabae933143fab0d0b6ae9014dbbc P c16c0dcf19b2426e8d54a191b46cf973: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-2/wal/wals/871eabae933143fab0d0b6ae9014dbbc
I20260927 03:28:08.294921 2745 ts_tablet_manager.cc:1939] T 871eabae933143fab0d0b6ae9014dbbc P 74c74bf308644ca1a7c3d6715a1d504f: Deleting consensus metadata
I20260927 03:28:08.294965 2746 ts_tablet_manager.cc:1939] T 871eabae933143fab0d0b6ae9014dbbc P c16c0dcf19b2426e8d54a191b46cf973: Deleting consensus metadata
I20260927 03:28:08.295277 2744 ts_tablet_manager.cc:1918] T 871eabae933143fab0d0b6ae9014dbbc P 5d9947006e7c4ae89d95618bb8ed6a48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 03:28:08.295295 2130 catalog_manager.cc:4841] TS c16c0dcf19b2426e8d54a191b46cf973 (127.0.98.3:42561): tablet 871eabae933143fab0d0b6ae9014dbbc (table test-table [id=6c5bdd94635641c78a26080c560509d5]) successfully deleted
I20260927 03:28:08.295361 2744 log.cc:1196] T 871eabae933143fab0d0b6ae9014dbbc P 5d9947006e7c4ae89d95618bb8ed6a48: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-0/wal/wals/871eabae933143fab0d0b6ae9014dbbc
I20260927 03:28:08.295382 2130 catalog_manager.cc:4841] TS 74c74bf308644ca1a7c3d6715a1d504f (127.0.98.2:42071): tablet 871eabae933143fab0d0b6ae9014dbbc (table test-table [id=6c5bdd94635641c78a26080c560509d5]) successfully deleted
I20260927 03:28:08.295603 2744 ts_tablet_manager.cc:1939] T 871eabae933143fab0d0b6ae9014dbbc P 5d9947006e7c4ae89d95618bb8ed6a48: Deleting consensus metadata
I20260927 03:28:08.296067 2127 catalog_manager.cc:4841] TS 5d9947006e7c4ae89d95618bb8ed6a48 (127.0.98.1:41697): tablet 871eabae933143fab0d0b6ae9014dbbc (table test-table [id=6c5bdd94635641c78a26080c560509d5]) successfully deleted
Sep 27 03:28:08 dist-test-slave-j0hf krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479688, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 03:28:08 dist-test-slave-j0hf krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479688, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:08.317759 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:08.309991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48438 (local address 127.0.98.62:42161)
0927 03:28:08.310228 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:08.310231 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:08.310317 (+ 86us) server_negotiation.cc:407] Connection header received
0927 03:28:08.310407 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:08.310408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:08.310431 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:08.310514 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:08.311132 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:08.311719 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:08.312503 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:08.312711 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:08.312797 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:08.315328 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:08.315329 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:08.315330 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:08.315337 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:08.316815 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:08.317133 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:08.317134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:08.317135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:08.317185 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:08.317358 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:08.317359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:08.317359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:08.317464 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:08.317545 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:08.317622 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 03:28:08.317683 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":69,"threads_started":1}
I20260927 03:28:08.319244 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:28:08.319407 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:28:08.320680 2280 tablet_service.cc:1467] Processing CreateTablet for tablet 78939040414545aaa07f96845a748bdd (DEFAULT_TABLE table=test-table [id=fa3c0e7f26454fb09daa4b7dd813a170]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:08.320731 2550 tablet_service.cc:1467] Processing CreateTablet for tablet 78939040414545aaa07f96845a748bdd (DEFAULT_TABLE table=test-table [id=fa3c0e7f26454fb09daa4b7dd813a170]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:08.320822 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78939040414545aaa07f96845a748bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:08.320827 2550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78939040414545aaa07f96845a748bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:08.322242 2420 tablet_service.cc:1467] Processing CreateTablet for tablet 78939040414545aaa07f96845a748bdd (DEFAULT_TABLE table=test-table [id=fa3c0e7f26454fb09daa4b7dd813a170]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:08.322319 2756 tablet_bootstrap.cc:492] T 78939040414545aaa07f96845a748bdd P 5d9947006e7c4ae89d95618bb8ed6a48: Bootstrap starting.
I20260927 03:28:08.322382 2420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78939040414545aaa07f96845a748bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:08.322319 2755 tablet_bootstrap.cc:492] T 78939040414545aaa07f96845a748bdd P c16c0dcf19b2426e8d54a191b46cf973: Bootstrap starting.
I20260927 03:28:08.322734 2755 tablet_bootstrap.cc:654] T 78939040414545aaa07f96845a748bdd P c16c0dcf19b2426e8d54a191b46cf973: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:08.322734 2756 tablet_bootstrap.cc:654] T 78939040414545aaa07f96845a748bdd P 5d9947006e7c4ae89d95618bb8ed6a48: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:08.323622 2755 tablet_bootstrap.cc:492] T 78939040414545aaa07f96845a748bdd P c16c0dcf19b2426e8d54a191b46cf973: No bootstrap required, opened a new log
I20260927 03:28:08.323622 2756 tablet_bootstrap.cc:492] T 78939040414545aaa07f96845a748bdd P 5d9947006e7c4ae89d95618bb8ed6a48: No bootstrap required, opened a new log
I20260927 03:28:08.323696 2755 ts_tablet_manager.cc:1397] T 78939040414545aaa07f96845a748bdd P c16c0dcf19b2426e8d54a191b46cf973: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:08.323696 2756 ts_tablet_manager.cc:1397] T 78939040414545aaa07f96845a748bdd P 5d9947006e7c4ae89d95618bb8ed6a48: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:08.323863 2759 tablet_bootstrap.cc:492] T 78939040414545aaa07f96845a748bdd P 74c74bf308644ca1a7c3d6715a1d504f: Bootstrap starting.
I20260927 03:28:08.323848 2755 raft_consensus.cc:348] T 78939040414545aaa07f96845a748bdd P c16c0dcf19b2426e8d54a191b46cf973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:08.323918 2755 raft_consensus.cc:374] T 78939040414545aaa07f96845a748bdd P c16c0dcf19b2426e8d54a191b46cf973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:08.323973 2755 raft_consensus.cc:729] T 78939040414545aaa07f96845a748bdd P c16c0dcf19b2426e8d54a191b46cf973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c16c0dcf19b2426e8d54a191b46cf973, State: Initialized, Role: FOLLOWER
I20260927 03:28:08.324049 2755 consensus_queue.cc:260] T 78939040414545aaa07f96845a748bdd P c16c0dcf19b2426e8d54a191b46cf973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:08.323927 2756 raft_consensus.cc:348] T 78939040414545aaa07f96845a748bdd P 5d9947006e7c4ae89d95618bb8ed6a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:08.324209 2756 raft_consensus.cc:374] T 78939040414545aaa07f96845a748bdd P 5d9947006e7c4ae89d95618bb8ed6a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:08.324273 2755 ts_tablet_manager.cc:1428] T 78939040414545aaa07f96845a748bdd P c16c0dcf19b2426e8d54a191b46cf973: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:08.324294 2759 tablet_bootstrap.cc:654] T 78939040414545aaa07f96845a748bdd P 74c74bf308644ca1a7c3d6715a1d504f: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:08.324301 2756 raft_consensus.cc:729] T 78939040414545aaa07f96845a748bdd P 5d9947006e7c4ae89d95618bb8ed6a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d9947006e7c4ae89d95618bb8ed6a48, State: Initialized, Role: FOLLOWER
I20260927 03:28:08.324384 2756 consensus_queue.cc:260] T 78939040414545aaa07f96845a748bdd P 5d9947006e7c4ae89d95618bb8ed6a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:08.324651 2756 ts_tablet_manager.cc:1428] T 78939040414545aaa07f96845a748bdd P 5d9947006e7c4ae89d95618bb8ed6a48: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 03:28:08.325029 2759 tablet_bootstrap.cc:492] T 78939040414545aaa07f96845a748bdd P 74c74bf308644ca1a7c3d6715a1d504f: No bootstrap required, opened a new log
I20260927 03:28:08.325104 2759 ts_tablet_manager.cc:1397] T 78939040414545aaa07f96845a748bdd P 74c74bf308644ca1a7c3d6715a1d504f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:08.325220 2759 raft_consensus.cc:348] T 78939040414545aaa07f96845a748bdd P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:08.325286 2759 raft_consensus.cc:374] T 78939040414545aaa07f96845a748bdd P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:08.325340 2759 raft_consensus.cc:729] T 78939040414545aaa07f96845a748bdd P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c74bf308644ca1a7c3d6715a1d504f, State: Initialized, Role: FOLLOWER
I20260927 03:28:08.325392 2759 consensus_queue.cc:260] T 78939040414545aaa07f96845a748bdd P 74c74bf308644ca1a7c3d6715a1d504f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:08.325489 2759 ts_tablet_manager.cc:1428] T 78939040414545aaa07f96845a748bdd P 74c74bf308644ca1a7c3d6715a1d504f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:08.540814 2760 raft_consensus.cc:482] T 78939040414545aaa07f96845a748bdd P c16c0dcf19b2426e8d54a191b46cf973 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:28:08.540927 2760 raft_consensus.cc:504] T 78939040414545aaa07f96845a748bdd P c16c0dcf19b2426e8d54a191b46cf973 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:08.541292 2760 leader_election.cc:288] T 78939040414545aaa07f96845a748bdd P c16c0dcf19b2426e8d54a191b46cf973 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d9947006e7c4ae89d95618bb8ed6a48 (127.0.98.1:41697), 74c74bf308644ca1a7c3d6715a1d504f (127.0.98.2:42071)
I20260927 03:28:08.545114 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:08.541600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54121 (local address 127.0.98.2:42071)
0927 03:28:08.541733 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:08.541735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:08.541813 (+ 78us) server_negotiation.cc:407] Connection header received
0927 03:28:08.541880 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:08.541881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:08.541894 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:08.541976 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:08.542112 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:08.542604 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:08.544152 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:08.544756 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:08.544825 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:08.544836 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:08.544994 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 03:28:08.545036 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":73,"threads_started":1}
I20260927 03:28:08.545123 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:08.541565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53525 (local address 127.0.98.1:41697)
0927 03:28:08.541705 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:08.541707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:08.541760 (+ 53us) server_negotiation.cc:407] Connection header received
0927 03:28:08.541851 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:08.541852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:08.541883 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:08.541954 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:08.542120 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:08.542598 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:08.544179 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:08.544755 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:08.544826 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:08.544836 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:08.545002 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 03:28:08.545039 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
I20260927 03:28:08.545171 2762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:08.541493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41697 (local address 127.0.98.3:53525)
0927 03:28:08.541701 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:08.541712 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:08.541777 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:08.541970 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:08.541970 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:08.541973 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:08.542037 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:08.542044 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:08.542720 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:08.542721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:08.544050 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:08.544064 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:08.544845 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:08.544847 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:08.544931 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:08.544933 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:28:08.544983 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 03:28:08.545026 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":34,"thread_start_us":145,"threads_started":1}
I20260927 03:28:08.545171 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:08.541507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42071 (local address 127.0.98.3:54121)
0927 03:28:08.541733 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:08.541743 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:08.541814 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:08.541979 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:08.541980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:08.541982 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:08.542031 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:08.542033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:08.542702 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:08.542703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:08.544036 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:08.544039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:08.544848 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:08.544848 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:08.544919 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:08.544925 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 03:28:08.544983 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 03:28:08.545026 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":98,"thread_start_us":129,"threads_started":1}
I20260927 03:28:08.545444 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78939040414545aaa07f96845a748bdd" candidate_uuid: "c16c0dcf19b2426e8d54a191b46cf973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" is_pre_election: true
I20260927 03:28:08.545552 2300 raft_consensus.cc:2454] T 78939040414545aaa07f96845a748bdd P 5d9947006e7c4ae89d95618bb8ed6a48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c16c0dcf19b2426e8d54a191b46cf973 in term 0.
I20260927 03:28:08.545519 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78939040414545aaa07f96845a748bdd" candidate_uuid: "c16c0dcf19b2426e8d54a191b46cf973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c74bf308644ca1a7c3d6715a1d504f" is_pre_election: true
I20260927 03:28:08.545598 2440 raft_consensus.cc:2454] T 78939040414545aaa07f96845a748bdd P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c16c0dcf19b2426e8d54a191b46cf973 in term 0.
I20260927 03:28:08.545754 2514 leader_election.cc:302] T 78939040414545aaa07f96845a748bdd P c16c0dcf19b2426e8d54a191b46cf973 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74c74bf308644ca1a7c3d6715a1d504f, c16c0dcf19b2426e8d54a191b46cf973; no voters:
I20260927 03:28:08.545907 2760 raft_consensus.cc:2781] T 78939040414545aaa07f96845a748bdd P c16c0dcf19b2426e8d54a191b46cf973 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:28:08.545962 2760 raft_consensus.cc:482] T 78939040414545aaa07f96845a748bdd P c16c0dcf19b2426e8d54a191b46cf973 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:28:08.545994 2760 raft_consensus.cc:3037] T 78939040414545aaa07f96845a748bdd P c16c0dcf19b2426e8d54a191b46cf973 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:08.546658 2760 raft_consensus.cc:504] T 78939040414545aaa07f96845a748bdd P c16c0dcf19b2426e8d54a191b46cf973 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:08.546797 2760 leader_election.cc:288] T 78939040414545aaa07f96845a748bdd P c16c0dcf19b2426e8d54a191b46cf973 [CANDIDATE]: Term 1 election: Requested vote from peers 5d9947006e7c4ae89d95618bb8ed6a48 (127.0.98.1:41697), 74c74bf308644ca1a7c3d6715a1d504f (127.0.98.2:42071)
I20260927 03:28:08.546926 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78939040414545aaa07f96845a748bdd" candidate_uuid: "c16c0dcf19b2426e8d54a191b46cf973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d9947006e7c4ae89d95618bb8ed6a48"
I20260927 03:28:08.546944 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78939040414545aaa07f96845a748bdd" candidate_uuid: "c16c0dcf19b2426e8d54a191b46cf973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c74bf308644ca1a7c3d6715a1d504f"
I20260927 03:28:08.547013 2300 raft_consensus.cc:3037] T 78939040414545aaa07f96845a748bdd P 5d9947006e7c4ae89d95618bb8ed6a48 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:08.547014 2440 raft_consensus.cc:3037] T 78939040414545aaa07f96845a748bdd P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:08.547843 2440 raft_consensus.cc:2454] T 78939040414545aaa07f96845a748bdd P 74c74bf308644ca1a7c3d6715a1d504f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c16c0dcf19b2426e8d54a191b46cf973 in term 1.
I20260927 03:28:08.547850 2300 raft_consensus.cc:2454] T 78939040414545aaa07f96845a748bdd P 5d9947006e7c4ae89d95618bb8ed6a48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c16c0dcf19b2426e8d54a191b46cf973 in term 1.
I20260927 03:28:08.547991 2514 leader_election.cc:302] T 78939040414545aaa07f96845a748bdd P c16c0dcf19b2426e8d54a191b46cf973 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74c74bf308644ca1a7c3d6715a1d504f, c16c0dcf19b2426e8d54a191b46cf973; no voters:
I20260927 03:28:08.548075 2760 raft_consensus.cc:2781] T 78939040414545aaa07f96845a748bdd P c16c0dcf19b2426e8d54a191b46cf973 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:08.548218 2760 raft_consensus.cc:686] T 78939040414545aaa07f96845a748bdd P c16c0dcf19b2426e8d54a191b46cf973 [term 1 LEADER]: Becoming Leader. State: Replica: c16c0dcf19b2426e8d54a191b46cf973, State: Running, Role: LEADER
I20260927 03:28:08.548297 2760 consensus_queue.cc:237] T 78939040414545aaa07f96845a748bdd P c16c0dcf19b2426e8d54a191b46cf973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:08.549006 2143 catalog_manager.cc:5495] T 78939040414545aaa07f96845a748bdd P c16c0dcf19b2426e8d54a191b46cf973 reported cstate change: term changed from 0 to 1, leader changed from <none> to c16c0dcf19b2426e8d54a191b46cf973 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c16c0dcf19b2426e8d54a191b46cf973" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } health_report { overall_health: HEALTHY } } }
I20260927 03:28:08.610493 2769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:08.607850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41900 (local address 127.0.98.3:42561)
0927 03:28:08.608024 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:08.608027 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:08.608037 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:28:08.608091 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:08.608091 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:08.608113 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:08.608204 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:28:08.608309 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:08.608878 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:08.609681 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:08.609907 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:08.609976 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:08.610178 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:28:08.610291 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:28:08.610356 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:08.610397 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 03:28:08.610415 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":119,"threads_started":1}
I20260927 03:28:08.614939 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:08.612971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45989 (local address 127.0.98.2:42071)
0927 03:28:08.613030 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:08.613032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:08.613079 (+ 47us) server_negotiation.cc:407] Connection header received
0927 03:28:08.613101 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:08.613102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:08.613112 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:08.613161 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:28:08.613264 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:08.613689 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:08.614395 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:08.614565 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:08.614611 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:08.614748 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:28:08.614772 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:28:08.614808 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:08.614873 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 03:28:08.614889 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 03:28:08.614946 2772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:08.612830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42071 (local address 127.0.98.3:45989)
0927 03:28:08.613013 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:08.613022 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:08.613070 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:08.613165 (+ 95us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:08.613166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:08.613167 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:28:08.613215 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:08.613215 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:08.613747 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:08.613748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:08.614308 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:08.614309 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:08.614625 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:08.614626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:08.614670 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:08.614681 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:28:08.614819 (+ 138us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:28:08.614821 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:28:08.614853 (+ 32us) client_negotiation.cc:240] Negotiation successful
0927 03:28:08.614884 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":94,"threads_started":1}
I20260927 03:28:08.617087 2440 raft_consensus.cc:1261] T 78939040414545aaa07f96845a748bdd P 74c74bf308644ca1a7c3d6715a1d504f [term 1 FOLLOWER]: Refusing update from remote peer c16c0dcf19b2426e8d54a191b46cf973: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:08.617128 2300 raft_consensus.cc:1261] T 78939040414545aaa07f96845a748bdd P 5d9947006e7c4ae89d95618bb8ed6a48 [term 1 FOLLOWER]: Refusing update from remote peer c16c0dcf19b2426e8d54a191b46cf973: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:08.617338 2766 consensus_queue.cc:1035] T 78939040414545aaa07f96845a748bdd P c16c0dcf19b2426e8d54a191b46cf973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:08.617455 2760 consensus_queue.cc:1035] T 78939040414545aaa07f96845a748bdd P c16c0dcf19b2426e8d54a191b46cf973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:08.625658 2769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:08.623097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47703 (local address 127.0.98.3:42561)
0927 03:28:08.623137 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:08.623139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:08.623305 (+ 166us) server_negotiation.cc:407] Connection header received
0927 03:28:08.623324 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:08.623325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:08.623340 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:08.623380 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:28:08.623550 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:08.624092 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:08.625083 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:08.625326 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:08.625376 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:08.625495 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:28:08.625521 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:28:08.625557 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:08.625586 (+ 29us) server_negotiation.cc:299] Negotiation successful
0927 03:28:08.625602 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:28:08.625710 2777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:08.623063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42561 (local address 127.0.98.2:47703)
0927 03:28:08.623194 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:08.623206 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:08.623260 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:08.623391 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:08.623396 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:08.623398 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:28:08.623488 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:08.623489 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:08.624157 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:08.624158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:08.625002 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:08.625004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:08.625390 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:08.625390 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:08.625432 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:08.625435 (+ 3us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:28:08.625555 (+ 120us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:28:08.625556 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 03:28:08.625585 (+ 29us) client_negotiation.cc:240] Negotiation successful
0927 03:28:08.625617 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":83,"threads_started":1}
I20260927 03:28:08.632200 2142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:28:08.632319 2142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:28:08.632941 2142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80fc0bdfbbfc4e259a2ac9cb3841e2d5: Sending DeleteTablet for 3 replicas of tablet 78939040414545aaa07f96845a748bdd
I20260927 03:28:08.633212 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet 78939040414545aaa07f96845a748bdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42052
I20260927 03:28:08.633225 2420 tablet_service.cc:1514] Processing DeleteTablet for tablet 78939040414545aaa07f96845a748bdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58852
I20260927 03:28:08.633306 2550 tablet_service.cc:1514] Processing DeleteTablet for tablet 78939040414545aaa07f96845a748bdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39962
I20260927 03:28:08.633325 2744 tablet_replica.cc:330] T 78939040414545aaa07f96845a748bdd P 5d9947006e7c4ae89d95618bb8ed6a48: stopping tablet replica
I20260927 03:28:08.633381 2744 raft_consensus.cc:2229] T 78939040414545aaa07f96845a748bdd P 5d9947006e7c4ae89d95618bb8ed6a48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:28:08.633404 2745 tablet_replica.cc:330] T 78939040414545aaa07f96845a748bdd P 74c74bf308644ca1a7c3d6715a1d504f: stopping tablet replica
I20260927 03:28:08.633417 2744 raft_consensus.cc:2258] T 78939040414545aaa07f96845a748bdd P 5d9947006e7c4ae89d95618bb8ed6a48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:08.633462 2745 raft_consensus.cc:2229] T 78939040414545aaa07f96845a748bdd P 74c74bf308644ca1a7c3d6715a1d504f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:28:08.633477 2746 tablet_replica.cc:330] T 78939040414545aaa07f96845a748bdd P c16c0dcf19b2426e8d54a191b46cf973: stopping tablet replica
I20260927 03:28:08.633500 2745 raft_consensus.cc:2258] T 78939040414545aaa07f96845a748bdd P 74c74bf308644ca1a7c3d6715a1d504f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:08.633526 2746 raft_consensus.cc:2229] T 78939040414545aaa07f96845a748bdd P c16c0dcf19b2426e8d54a191b46cf973 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:28:08.633605 2746 raft_consensus.cc:2258] T 78939040414545aaa07f96845a748bdd P c16c0dcf19b2426e8d54a191b46cf973 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:08.633839 2745 ts_tablet_manager.cc:1905] T 78939040414545aaa07f96845a748bdd P 74c74bf308644ca1a7c3d6715a1d504f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:08.634019 2744 ts_tablet_manager.cc:1905] T 78939040414545aaa07f96845a748bdd P 5d9947006e7c4ae89d95618bb8ed6a48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:08.635617 2746 ts_tablet_manager.cc:1905] T 78939040414545aaa07f96845a748bdd P c16c0dcf19b2426e8d54a191b46cf973: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:08.635613 2744 ts_tablet_manager.cc:1918] T 78939040414545aaa07f96845a748bdd P 5d9947006e7c4ae89d95618bb8ed6a48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:28:08.635666 2744 log.cc:1196] T 78939040414545aaa07f96845a748bdd P 5d9947006e7c4ae89d95618bb8ed6a48: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-0/wal/wals/78939040414545aaa07f96845a748bdd
I20260927 03:28:08.635893 2744 ts_tablet_manager.cc:1939] T 78939040414545aaa07f96845a748bdd P 5d9947006e7c4ae89d95618bb8ed6a48: Deleting consensus metadata
I20260927 03:28:08.636998 2127 catalog_manager.cc:4841] TS 5d9947006e7c4ae89d95618bb8ed6a48 (127.0.98.1:41697): tablet 78939040414545aaa07f96845a748bdd (table test-table [id=fa3c0e7f26454fb09daa4b7dd813a170]) successfully deleted
I20260927 03:28:08.637142 2142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:28:08.637300 2142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:28:08.637539 2746 ts_tablet_manager.cc:1918] T 78939040414545aaa07f96845a748bdd P c16c0dcf19b2426e8d54a191b46cf973: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:28:08.637596 2746 log.cc:1196] T 78939040414545aaa07f96845a748bdd P c16c0dcf19b2426e8d54a191b46cf973: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-2/wal/wals/78939040414545aaa07f96845a748bdd
I20260927 03:28:08.637809 2746 ts_tablet_manager.cc:1939] T 78939040414545aaa07f96845a748bdd P c16c0dcf19b2426e8d54a191b46cf973: Deleting consensus metadata
I20260927 03:28:08.638067 2130 catalog_manager.cc:4841] TS c16c0dcf19b2426e8d54a191b46cf973 (127.0.98.3:42561): tablet 78939040414545aaa07f96845a748bdd (table test-table [id=fa3c0e7f26454fb09daa4b7dd813a170]) successfully deleted
I20260927 03:28:08.638113 2745 ts_tablet_manager.cc:1918] T 78939040414545aaa07f96845a748bdd P 74c74bf308644ca1a7c3d6715a1d504f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:28:08.638157 2745 log.cc:1196] T 78939040414545aaa07f96845a748bdd P 74c74bf308644ca1a7c3d6715a1d504f: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-1/wal/wals/78939040414545aaa07f96845a748bdd
I20260927 03:28:08.638360 2745 ts_tablet_manager.cc:1939] T 78939040414545aaa07f96845a748bdd P 74c74bf308644ca1a7c3d6715a1d504f: Deleting consensus metadata
I20260927 03:28:08.638630 2550 tablet_service.cc:1467] Processing CreateTablet for tablet 8181d5f591f9495ab39900f3ed193b3a (DEFAULT_TABLE table=test-table [id=2bfe34ea2db741b1a77fc66897aaf332]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:08.638773 2550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8181d5f591f9495ab39900f3ed193b3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:08.638772 2280 tablet_service.cc:1467] Processing CreateTablet for tablet 8181d5f591f9495ab39900f3ed193b3a (DEFAULT_TABLE table=test-table [id=2bfe34ea2db741b1a77fc66897aaf332]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:08.638888 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8181d5f591f9495ab39900f3ed193b3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:08.639787 2755 tablet_bootstrap.cc:492] T 8181d5f591f9495ab39900f3ed193b3a P c16c0dcf19b2426e8d54a191b46cf973: Bootstrap starting.
I20260927 03:28:08.640026 2130 catalog_manager.cc:4841] TS 74c74bf308644ca1a7c3d6715a1d504f (127.0.98.2:42071): tablet 78939040414545aaa07f96845a748bdd (table test-table [id=fa3c0e7f26454fb09daa4b7dd813a170]) successfully deleted
I20260927 03:28:08.640153 2420 tablet_service.cc:1467] Processing CreateTablet for tablet 8181d5f591f9495ab39900f3ed193b3a (DEFAULT_TABLE table=test-table [id=2bfe34ea2db741b1a77fc66897aaf332]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:08.640187 2755 tablet_bootstrap.cc:654] T 8181d5f591f9495ab39900f3ed193b3a P c16c0dcf19b2426e8d54a191b46cf973: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:08.640247 2420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8181d5f591f9495ab39900f3ed193b3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:08.640745 2755 tablet_bootstrap.cc:492] T 8181d5f591f9495ab39900f3ed193b3a P c16c0dcf19b2426e8d54a191b46cf973: No bootstrap required, opened a new log
I20260927 03:28:08.640810 2755 ts_tablet_manager.cc:1397] T 8181d5f591f9495ab39900f3ed193b3a P c16c0dcf19b2426e8d54a191b46cf973: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:08.640981 2755 raft_consensus.cc:348] T 8181d5f591f9495ab39900f3ed193b3a P c16c0dcf19b2426e8d54a191b46cf973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:08.641028 2755 raft_consensus.cc:374] T 8181d5f591f9495ab39900f3ed193b3a P c16c0dcf19b2426e8d54a191b46cf973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:08.641041 2755 raft_consensus.cc:729] T 8181d5f591f9495ab39900f3ed193b3a P c16c0dcf19b2426e8d54a191b46cf973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c16c0dcf19b2426e8d54a191b46cf973, State: Initialized, Role: FOLLOWER
I20260927 03:28:08.641096 2755 consensus_queue.cc:260] T 8181d5f591f9495ab39900f3ed193b3a P c16c0dcf19b2426e8d54a191b46cf973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:08.641196 2755 ts_tablet_manager.cc:1428] T 8181d5f591f9495ab39900f3ed193b3a P c16c0dcf19b2426e8d54a191b46cf973: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:08.641423 2756 tablet_bootstrap.cc:492] T 8181d5f591f9495ab39900f3ed193b3a P 5d9947006e7c4ae89d95618bb8ed6a48: Bootstrap starting.
I20260927 03:28:08.641722 2756 tablet_bootstrap.cc:654] T 8181d5f591f9495ab39900f3ed193b3a P 5d9947006e7c4ae89d95618bb8ed6a48: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:08.641949 2759 tablet_bootstrap.cc:492] T 8181d5f591f9495ab39900f3ed193b3a P 74c74bf308644ca1a7c3d6715a1d504f: Bootstrap starting.
I20260927 03:28:08.642211 2759 tablet_bootstrap.cc:654] T 8181d5f591f9495ab39900f3ed193b3a P 74c74bf308644ca1a7c3d6715a1d504f: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:08.642653 2756 tablet_bootstrap.cc:492] T 8181d5f591f9495ab39900f3ed193b3a P 5d9947006e7c4ae89d95618bb8ed6a48: No bootstrap required, opened a new log
I20260927 03:28:08.642695 2756 ts_tablet_manager.cc:1397] T 8181d5f591f9495ab39900f3ed193b3a P 5d9947006e7c4ae89d95618bb8ed6a48: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 03:28:08.642762 2759 tablet_bootstrap.cc:492] T 8181d5f591f9495ab39900f3ed193b3a P 74c74bf308644ca1a7c3d6715a1d504f: No bootstrap required, opened a new log
I20260927 03:28:08.642796 2759 ts_tablet_manager.cc:1397] T 8181d5f591f9495ab39900f3ed193b3a P 74c74bf308644ca1a7c3d6715a1d504f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 03:28:08.642798 2756 raft_consensus.cc:348] T 8181d5f591f9495ab39900f3ed193b3a P 5d9947006e7c4ae89d95618bb8ed6a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:08.642875 2756 raft_consensus.cc:374] T 8181d5f591f9495ab39900f3ed193b3a P 5d9947006e7c4ae89d95618bb8ed6a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:08.642896 2756 raft_consensus.cc:729] T 8181d5f591f9495ab39900f3ed193b3a P 5d9947006e7c4ae89d95618bb8ed6a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d9947006e7c4ae89d95618bb8ed6a48, State: Initialized, Role: FOLLOWER
I20260927 03:28:08.642895 2759 raft_consensus.cc:348] T 8181d5f591f9495ab39900f3ed193b3a P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:08.642941 2759 raft_consensus.cc:374] T 8181d5f591f9495ab39900f3ed193b3a P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:08.642928 2756 consensus_queue.cc:260] T 8181d5f591f9495ab39900f3ed193b3a P 5d9947006e7c4ae89d95618bb8ed6a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:08.642954 2759 raft_consensus.cc:729] T 8181d5f591f9495ab39900f3ed193b3a P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c74bf308644ca1a7c3d6715a1d504f, State: Initialized, Role: FOLLOWER
I20260927 03:28:08.642992 2759 consensus_queue.cc:260] T 8181d5f591f9495ab39900f3ed193b3a P 74c74bf308644ca1a7c3d6715a1d504f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:08.643020 2756 ts_tablet_manager.cc:1428] T 8181d5f591f9495ab39900f3ed193b3a P 5d9947006e7c4ae89d95618bb8ed6a48: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:08.643090 2759 ts_tablet_manager.cc:1428] T 8181d5f591f9495ab39900f3ed193b3a P 74c74bf308644ca1a7c3d6715a1d504f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:08.693425 2705 raft_consensus.cc:482] T 8181d5f591f9495ab39900f3ed193b3a P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:28:08.693521 2705 raft_consensus.cc:504] T 8181d5f591f9495ab39900f3ed193b3a P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:08.693715 2705 leader_election.cc:288] T 8181d5f591f9495ab39900f3ed193b3a P 74c74bf308644ca1a7c3d6715a1d504f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d9947006e7c4ae89d95618bb8ed6a48 (127.0.98.1:41697), c16c0dcf19b2426e8d54a191b46cf973 (127.0.98.3:42561)
I20260927 03:28:08.693953 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8181d5f591f9495ab39900f3ed193b3a" candidate_uuid: "74c74bf308644ca1a7c3d6715a1d504f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" is_pre_election: true
I20260927 03:28:08.693972 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8181d5f591f9495ab39900f3ed193b3a" candidate_uuid: "74c74bf308644ca1a7c3d6715a1d504f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c16c0dcf19b2426e8d54a191b46cf973" is_pre_election: true
I20260927 03:28:08.694064 2300 raft_consensus.cc:2454] T 8181d5f591f9495ab39900f3ed193b3a P 5d9947006e7c4ae89d95618bb8ed6a48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74c74bf308644ca1a7c3d6715a1d504f in term 0.
I20260927 03:28:08.694074 2580 raft_consensus.cc:2454] T 8181d5f591f9495ab39900f3ed193b3a P c16c0dcf19b2426e8d54a191b46cf973 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74c74bf308644ca1a7c3d6715a1d504f in term 0.
I20260927 03:28:08.694242 2371 leader_election.cc:302] T 8181d5f591f9495ab39900f3ed193b3a P 74c74bf308644ca1a7c3d6715a1d504f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d9947006e7c4ae89d95618bb8ed6a48, 74c74bf308644ca1a7c3d6715a1d504f; no voters:
I20260927 03:28:08.694332 2705 raft_consensus.cc:2781] T 8181d5f591f9495ab39900f3ed193b3a P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:28:08.694365 2705 raft_consensus.cc:482] T 8181d5f591f9495ab39900f3ed193b3a P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:28:08.694386 2705 raft_consensus.cc:3037] T 8181d5f591f9495ab39900f3ed193b3a P 74c74bf308644ca1a7c3d6715a1d504f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:08.695322 2705 raft_consensus.cc:504] T 8181d5f591f9495ab39900f3ed193b3a P 74c74bf308644ca1a7c3d6715a1d504f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:08.695497 2705 leader_election.cc:288] T 8181d5f591f9495ab39900f3ed193b3a P 74c74bf308644ca1a7c3d6715a1d504f [CANDIDATE]: Term 1 election: Requested vote from peers 5d9947006e7c4ae89d95618bb8ed6a48 (127.0.98.1:41697), c16c0dcf19b2426e8d54a191b46cf973 (127.0.98.3:42561)
I20260927 03:28:08.695653 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8181d5f591f9495ab39900f3ed193b3a" candidate_uuid: "74c74bf308644ca1a7c3d6715a1d504f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d9947006e7c4ae89d95618bb8ed6a48"
I20260927 03:28:08.695667 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8181d5f591f9495ab39900f3ed193b3a" candidate_uuid: "74c74bf308644ca1a7c3d6715a1d504f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c16c0dcf19b2426e8d54a191b46cf973"
I20260927 03:28:08.695730 2300 raft_consensus.cc:3037] T 8181d5f591f9495ab39900f3ed193b3a P 5d9947006e7c4ae89d95618bb8ed6a48 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:08.695747 2580 raft_consensus.cc:3037] T 8181d5f591f9495ab39900f3ed193b3a P c16c0dcf19b2426e8d54a191b46cf973 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:08.696345 2580 raft_consensus.cc:2454] T 8181d5f591f9495ab39900f3ed193b3a P c16c0dcf19b2426e8d54a191b46cf973 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74c74bf308644ca1a7c3d6715a1d504f in term 1.
I20260927 03:28:08.696465 2300 raft_consensus.cc:2454] T 8181d5f591f9495ab39900f3ed193b3a P 5d9947006e7c4ae89d95618bb8ed6a48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74c74bf308644ca1a7c3d6715a1d504f in term 1.
I20260927 03:28:08.696527 2374 leader_election.cc:302] T 8181d5f591f9495ab39900f3ed193b3a P 74c74bf308644ca1a7c3d6715a1d504f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74c74bf308644ca1a7c3d6715a1d504f, c16c0dcf19b2426e8d54a191b46cf973; no voters:
I20260927 03:28:08.696645 2705 raft_consensus.cc:2781] T 8181d5f591f9495ab39900f3ed193b3a P 74c74bf308644ca1a7c3d6715a1d504f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:08.696687 2705 raft_consensus.cc:686] T 8181d5f591f9495ab39900f3ed193b3a P 74c74bf308644ca1a7c3d6715a1d504f [term 1 LEADER]: Becoming Leader. State: Replica: 74c74bf308644ca1a7c3d6715a1d504f, State: Running, Role: LEADER
I20260927 03:28:08.696743 2705 consensus_queue.cc:237] T 8181d5f591f9495ab39900f3ed193b3a P 74c74bf308644ca1a7c3d6715a1d504f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } }
I20260927 03:28:08.697450 2142 catalog_manager.cc:5495] T 8181d5f591f9495ab39900f3ed193b3a P 74c74bf308644ca1a7c3d6715a1d504f reported cstate change: term changed from 0 to 1, leader changed from <none> to 74c74bf308644ca1a7c3d6715a1d504f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "74c74bf308644ca1a7c3d6715a1d504f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74c74bf308644ca1a7c3d6715a1d504f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 } health_report { overall_health: UNKNOWN } } }
I20260927 03:28:08.708907 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:08.706429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38122 (local address 127.0.98.2:42071)
0927 03:28:08.706469 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:08.706471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:08.706480 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:28:08.706529 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:08.706530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:08.706540 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:08.706607 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:28:08.706688 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:08.707414 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:08.708150 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:08.708454 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:08.708528 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:08.708664 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:28:08.708699 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:28:08.708760 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:08.708795 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 03:28:08.708818 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:28:08.711184 2580 raft_consensus.cc:1261] T 8181d5f591f9495ab39900f3ed193b3a P c16c0dcf19b2426e8d54a191b46cf973 [term 1 FOLLOWER]: Refusing update from remote peer 74c74bf308644ca1a7c3d6715a1d504f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:08.711390 2725 consensus_queue.cc:1035] T 8181d5f591f9495ab39900f3ed193b3a P 74c74bf308644ca1a7c3d6715a1d504f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c16c0dcf19b2426e8d54a191b46cf973" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:08.711479 2300 raft_consensus.cc:1261] T 8181d5f591f9495ab39900f3ed193b3a P 5d9947006e7c4ae89d95618bb8ed6a48 [term 1 FOLLOWER]: Refusing update from remote peer 74c74bf308644ca1a7c3d6715a1d504f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:08.711647 2725 consensus_queue.cc:1035] T 8181d5f591f9495ab39900f3ed193b3a P 74c74bf308644ca1a7c3d6715a1d504f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d9947006e7c4ae89d95618bb8ed6a48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 03:28:08.716816 2723 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 03:28:10.242429 2142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:28:10.242553 2142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:28:10.243567 2142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80fc0bdfbbfc4e259a2ac9cb3841e2d5: Sending DeleteTablet for 3 replicas of tablet 8181d5f591f9495ab39900f3ed193b3a
I20260927 03:28:10.243903 2550 tablet_service.cc:1514] Processing DeleteTablet for tablet 8181d5f591f9495ab39900f3ed193b3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39962
I20260927 03:28:10.243911 2419 tablet_service.cc:1514] Processing DeleteTablet for tablet 8181d5f591f9495ab39900f3ed193b3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58852
I20260927 03:28:10.243911 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet 8181d5f591f9495ab39900f3ed193b3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42052
I20260927 03:28:10.244122 2807 tablet_replica.cc:330] T 8181d5f591f9495ab39900f3ed193b3a P c16c0dcf19b2426e8d54a191b46cf973: stopping tablet replica
I20260927 03:28:10.244189 2807 raft_consensus.cc:2229] T 8181d5f591f9495ab39900f3ed193b3a P c16c0dcf19b2426e8d54a191b46cf973 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:28:10.244240 2807 raft_consensus.cc:2258] T 8181d5f591f9495ab39900f3ed193b3a P c16c0dcf19b2426e8d54a191b46cf973 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:10.244216 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 2215
I20260927 03:28:10.245018 2807 ts_tablet_manager.cc:1905] T 8181d5f591f9495ab39900f3ed193b3a P c16c0dcf19b2426e8d54a191b46cf973: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:10.247169 2807 ts_tablet_manager.cc:1918] T 8181d5f591f9495ab39900f3ed193b3a P c16c0dcf19b2426e8d54a191b46cf973: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 03:28:10.247249 2807 log.cc:1196] T 8181d5f591f9495ab39900f3ed193b3a P c16c0dcf19b2426e8d54a191b46cf973: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-2/wal/wals/8181d5f591f9495ab39900f3ed193b3a
I20260927 03:28:10.247537 2807 ts_tablet_manager.cc:1939] T 8181d5f591f9495ab39900f3ed193b3a P c16c0dcf19b2426e8d54a191b46cf973: Deleting consensus metadata
I20260927 03:28:10.247931 2130 catalog_manager.cc:4841] TS c16c0dcf19b2426e8d54a191b46cf973 (127.0.98.3:42561): tablet 8181d5f591f9495ab39900f3ed193b3a (table test-table [id=2bfe34ea2db741b1a77fc66897aaf332]) successfully deleted
I20260927 03:28:10.251572 2806 tablet_replica.cc:330] T 8181d5f591f9495ab39900f3ed193b3a P 74c74bf308644ca1a7c3d6715a1d504f: stopping tablet replica
I20260927 03:28:10.251664 2806 raft_consensus.cc:2229] T 8181d5f591f9495ab39900f3ed193b3a P 74c74bf308644ca1a7c3d6715a1d504f [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:28:10.251767 2806 raft_consensus.cc:2258] T 8181d5f591f9495ab39900f3ed193b3a P 74c74bf308644ca1a7c3d6715a1d504f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:10.252084 2806 ts_tablet_manager.cc:1905] T 8181d5f591f9495ab39900f3ed193b3a P 74c74bf308644ca1a7c3d6715a1d504f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 03:28:10.254107 2127 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41697 (error 108)
I20260927 03:28:10.254179 2806 ts_tablet_manager.cc:1918] T 8181d5f591f9495ab39900f3ed193b3a P 74c74bf308644ca1a7c3d6715a1d504f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 03:28:10.254230 2806 log.cc:1196] T 8181d5f591f9495ab39900f3ed193b3a P 74c74bf308644ca1a7c3d6715a1d504f: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790479675865103-392-0/minicluster-data/ts-1/wal/wals/8181d5f591f9495ab39900f3ed193b3a
I20260927 03:28:10.254505 2806 ts_tablet_manager.cc:1939] T 8181d5f591f9495ab39900f3ed193b3a P 74c74bf308644ca1a7c3d6715a1d504f: Deleting consensus metadata
I20260927 03:28:10.254783 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 2355
I20260927 03:28:10.256178 2809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:10.254620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41697 (local address 127.0.0.1:52688)
0927 03:28:10.254783 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:10.254838 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":103,"threads_started":1}
W20260927 03:28:10.256323 2127 catalog_manager.cc:4584] TS 5d9947006e7c4ae89d95618bb8ed6a48 (127.0.98.1:41697): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8181d5f591f9495ab39900f3ed193b3a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41697: connect: Connection refused (error 111)
I20260927 03:28:10.261185 2809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:10.261045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42071 (local address 127.0.0.1:38134)
0927 03:28:10.261111 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:10.261144 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 03:28:10.261206 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 2495
I20260927 03:28:10.268421 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 2111
2026-09-27T03:28:10Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6363 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:28:10 dist-test-slave-j0hf krb5kdc[2812](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:28:10 dist-test-slave-j0hf krb5kdc[2812](info): set up 2 sockets
Sep 27 03:28:10 dist-test-slave-j0hf krb5kdc[2812](info): commencing operation
krb5kdc: starting...
W20260927 03:28:11.311167 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:28:11 dist-test-slave-j0hf krb5kdc[2812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:28:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:28:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:28:11.430595 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35403
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:11.511471 2825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:11.511716 2825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:11.511755 2825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:11.512928 2825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:28:11.513005 2825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:11.513031 2825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:11.513064 2825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:28:11.513089 2825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:28:11.514644 2825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:11.514925 2825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:11.515163 2825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:11.519932 2833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:11.519958 2830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:11.520038 2831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:11.520326 2825 server_base.cc:1034] running on GCE node
I20260927 03:28:11.520596 2825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:11.520896 2825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:11.522078 2825 hybrid_clock.cc:648] HybridClock initialized: now 1790479691522048 us; error 40 us; skew 500 ppm
Sep 27 03:28:11 dist-test-slave-j0hf krb5kdc[2812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:11.530614 2825 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:28:11.531075 2825 webserver.cc:460] Webserver started at http://127.0.98.62:36929/ using document root <none> and password file <none>
I20260927 03:28:11.531320 2825 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:11.531366 2825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:11.531566 2825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:11.532648 2825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "9227d25a309649dc8dda9ec0aa7c2f5b"
format_stamp: "Formatted at 2026-09-27 03:28:11 on dist-test-slave-j0hf"
server_key: "2f530a8379b9acf329ca8f9e4650c496"
server_key_iv: "527f7d230bce4707bb07d9bb7c0250bd"
server_key_version: "encryptionkey@0"
I20260927 03:28:11.533020 2825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "9227d25a309649dc8dda9ec0aa7c2f5b"
format_stamp: "Formatted at 2026-09-27 03:28:11 on dist-test-slave-j0hf"
server_key: "2f530a8379b9acf329ca8f9e4650c496"
server_key_iv: "527f7d230bce4707bb07d9bb7c0250bd"
server_key_version: "encryptionkey@0"
I20260927 03:28:11.534379 2825 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:28:11.535115 2840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:11.535288 2825 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:28:11.535399 2825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "9227d25a309649dc8dda9ec0aa7c2f5b"
format_stamp: "Formatted at 2026-09-27 03:28:11 on dist-test-slave-j0hf"
server_key: "2f530a8379b9acf329ca8f9e4650c496"
server_key_iv: "527f7d230bce4707bb07d9bb7c0250bd"
server_key_version: "encryptionkey@0"
I20260927 03:28:11.535530 2825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:11.564424 2825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:11.566428 2825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:11.566603 2825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:11.570665 2825 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35403
I20260927 03:28:11.570719 2892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35403 every 8 connection(s)
I20260927 03:28:11.571265 2825 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:28:11.571951 2893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:11.574170 2893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9227d25a309649dc8dda9ec0aa7c2f5b: Bootstrap starting.
I20260927 03:28:11.574690 2893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9227d25a309649dc8dda9ec0aa7c2f5b: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:11.574928 2893 log.cc:824] T 00000000000000000000000000000000 P 9227d25a309649dc8dda9ec0aa7c2f5b: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:11.575609 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 2825
I20260927 03:28:11.575652 2893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9227d25a309649dc8dda9ec0aa7c2f5b: No bootstrap required, opened a new log
I20260927 03:28:11.575711 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:28:11.575871 392 external_mini_cluster.cc:1428] Setting key 057920a9539386d903e0a5b46c7aeebc
I20260927 03:28:11.578163 2893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9227d25a309649dc8dda9ec0aa7c2f5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9227d25a309649dc8dda9ec0aa7c2f5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35403 } }
I20260927 03:28:11.578334 2893 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9227d25a309649dc8dda9ec0aa7c2f5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:11.578372 2893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9227d25a309649dc8dda9ec0aa7c2f5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9227d25a309649dc8dda9ec0aa7c2f5b, State: Initialized, Role: FOLLOWER
I20260927 03:28:11.578524 2893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9227d25a309649dc8dda9ec0aa7c2f5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9227d25a309649dc8dda9ec0aa7c2f5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35403 } }
I20260927 03:28:11.578589 2893 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9227d25a309649dc8dda9ec0aa7c2f5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:28:11.578610 2893 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9227d25a309649dc8dda9ec0aa7c2f5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:28:11.578666 2893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9227d25a309649dc8dda9ec0aa7c2f5b [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:11.579259 2893 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9227d25a309649dc8dda9ec0aa7c2f5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9227d25a309649dc8dda9ec0aa7c2f5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35403 } }
I20260927 03:28:11.579361 2893 leader_election.cc:302] T 00000000000000000000000000000000 P 9227d25a309649dc8dda9ec0aa7c2f5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9227d25a309649dc8dda9ec0aa7c2f5b; no voters:
I20260927 03:28:11.579540 2893 leader_election.cc:288] T 00000000000000000000000000000000 P 9227d25a309649dc8dda9ec0aa7c2f5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:28:11.579612 2898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9227d25a309649dc8dda9ec0aa7c2f5b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:11.579787 2893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9227d25a309649dc8dda9ec0aa7c2f5b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:28:11.579793 2898 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9227d25a309649dc8dda9ec0aa7c2f5b [term 1 LEADER]: Becoming Leader. State: Replica: 9227d25a309649dc8dda9ec0aa7c2f5b, State: Running, Role: LEADER
I20260927 03:28:11.580096 2898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9227d25a309649dc8dda9ec0aa7c2f5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9227d25a309649dc8dda9ec0aa7c2f5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35403 } }
I20260927 03:28:11.580479 2900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9227d25a309649dc8dda9ec0aa7c2f5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9227d25a309649dc8dda9ec0aa7c2f5b. Latest consensus state: current_term: 1 leader_uuid: "9227d25a309649dc8dda9ec0aa7c2f5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9227d25a309649dc8dda9ec0aa7c2f5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35403 } } }
I20260927 03:28:11.580703 2900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9227d25a309649dc8dda9ec0aa7c2f5b [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:11.581166 2907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:28:11.580490 2899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9227d25a309649dc8dda9ec0aa7c2f5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9227d25a309649dc8dda9ec0aa7c2f5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9227d25a309649dc8dda9ec0aa7c2f5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35403 } } }
I20260927 03:28:11.581453 2899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9227d25a309649dc8dda9ec0aa7c2f5b [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:11.581723 2907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:28:11.583346 2907 catalog_manager.cc:1347] Generated new cluster ID: 2b8f3bc8bb584d788412fa472e5de35d
I20260927 03:28:11.583389 2907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:28:11 dist-test-slave-j0hf krb5kdc[2812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:11.588939 2897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:11.576423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35444 (local address 127.0.98.62:35403)
0927 03:28:11.576639 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:11.576649 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:11.576687 (+ 38us) server_negotiation.cc:407] Connection header received
0927 03:28:11.578128 (+ 1441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:11.578138 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:11.578200 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:11.578363 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:11.579120 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:11.580663 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:11.581798 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:11.582397 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:11.582549 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:11.585358 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:11.585362 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:11.585362 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:11.585374 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:11.587264 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:11.587730 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:11.587733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:11.587733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:11.587847 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:11.588063 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:11.588064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:11.588064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:11.588346 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:11.588483 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:11.588617 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 03:28:11.588742 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":98,"threads_started":1}
I20260927 03:28:11.599189 2907 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:28:11.599797 2907 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:28:11.603612 2907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9227d25a309649dc8dda9ec0aa7c2f5b: Generated new TSK 0
I20260927 03:28:11.603811 2907 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:28:11.634025 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35403
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:11.717823 2919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:11.718087 2919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:11.718114 2919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:11.719254 2919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:11.719295 2919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:11.719323 2919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:28:11.720968 2919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:11.721180 2919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:11.721367 2919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:11.726449 2924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:11.726442 2925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:11.726442 2927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:11.726670 2919 server_base.cc:1034] running on GCE node
I20260927 03:28:11.726977 2919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:11.727227 2919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:11.728375 2919 hybrid_clock.cc:648] HybridClock initialized: now 1790479691728359 us; error 35 us; skew 500 ppm
Sep 27 03:28:11 dist-test-slave-j0hf krb5kdc[2812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:11.731088 2919 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:28:11.731673 2919 webserver.cc:460] Webserver started at http://127.0.98.1:46019/ using document root <none> and password file <none>
I20260927 03:28:11.731950 2919 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:11.731999 2919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:11.732151 2919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:11.733296 2919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/data/instance:
uuid: "c63cffab52e04ad7a59b569f0397aa93"
format_stamp: "Formatted at 2026-09-27 03:28:11 on dist-test-slave-j0hf"
server_key: "de42686fd61922a37a93f635ec4923d8"
server_key_iv: "10c138b727cbba2fc44ec7fc4245493c"
server_key_version: "encryptionkey@0"
I20260927 03:28:11.733676 2919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c63cffab52e04ad7a59b569f0397aa93"
format_stamp: "Formatted at 2026-09-27 03:28:11 on dist-test-slave-j0hf"
server_key: "de42686fd61922a37a93f635ec4923d8"
server_key_iv: "10c138b727cbba2fc44ec7fc4245493c"
server_key_version: "encryptionkey@0"
I20260927 03:28:11.735061 2919 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:28:11.735942 2934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:11.736129 2919 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:28:11.736258 2919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "c63cffab52e04ad7a59b569f0397aa93"
format_stamp: "Formatted at 2026-09-27 03:28:11 on dist-test-slave-j0hf"
server_key: "de42686fd61922a37a93f635ec4923d8"
server_key_iv: "10c138b727cbba2fc44ec7fc4245493c"
server_key_version: "encryptionkey@0"
I20260927 03:28:11.736334 2919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:11.760462 2919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:11.762538 2919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:11.762717 2919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:11.762989 2919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:11.763324 2919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:11.763358 2919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:11.763381 2919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:11.763453 2919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:11.769258 2919 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33523
I20260927 03:28:11.769313 3047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33523 every 8 connection(s)
I20260927 03:28:11.769677 2919 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:28:11 dist-test-slave-j0hf krb5kdc[2812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:11.779016 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 2919
I20260927 03:28:11.779110 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:28:11.779307 392 external_mini_cluster.cc:1428] Setting key f4684245fc33088950b9dc1fc66309f2
I20260927 03:28:11.779837 2897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:11.770444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60677 (local address 127.0.98.62:35403)
0927 03:28:11.770536 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:11.770539 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:11.770734 (+ 195us) server_negotiation.cc:407] Connection header received
0927 03:28:11.771799 (+ 1065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:11.771801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:11.771814 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:11.771868 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:11.772606 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:11.773132 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:11.774177 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:11.774409 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:11.774499 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:11.776818 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:11.776819 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:11.776820 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:11.776827 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:11.778274 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:11.778696 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:11.778697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:11.778697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:11.778763 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:11.779030 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:11.779031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:11.779031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:11.779203 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:11.779281 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:11.779632 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 03:28:11.779723 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 03:28:11.780105 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:11.770405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35403 (local address 127.0.98.1:60677)
0927 03:28:11.770601 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:11.770640 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:11.771680 (+ 1040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:11.771957 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:11.771960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:11.772297 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:11.772495 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:11.772499 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:11.773232 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:11.773233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:11.774076 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:11.774079 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:11.774512 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:11.774513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:11.774636 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:11.775231 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:11.775241 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:11.776709 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:11.778381 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:11.778389 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:11.778391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:11.778618 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:11.778847 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:11.778848 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:11.778848 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:11.778960 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:11.779273 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:11.779280 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:11.779483 (+ 203us) client_negotiation.cc:769] Sending connection context
0927 03:28:11.779612 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 03:28:11.779837 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":106,"threads_started":1}
I20260927 03:28:11.780539 3048 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35403
I20260927 03:28:11.780654 3048 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:11.780865 3048 heartbeater.cc:511] Master 127.0.98.62:35403 requested a full tablet report, sending...
I20260927 03:28:11.781394 2857 ts_manager.cc:194] Registered new tserver with Master: c63cffab52e04ad7a59b569f0397aa93 (127.0.98.1:33523)
I20260927 03:28:11.782048 2857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60677
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:28:11.806386 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35403
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:11.888571 3053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:11.888835 3053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:11.888875 3053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:11.890069 3053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:11.890146 3053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:11.890190 3053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:28:11.891762 3053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:11.892045 3053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:11.892293 3053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:11.897377 3059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:11.897354 3058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:11.897444 3061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:11.897678 3053 server_base.cc:1034] running on GCE node
I20260927 03:28:11.897917 3053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:11.898151 3053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:11.899324 3053 hybrid_clock.cc:648] HybridClock initialized: now 1790479691899317 us; error 37 us; skew 500 ppm
Sep 27 03:28:11 dist-test-slave-j0hf krb5kdc[2812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:11.903017 3053 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:28:11.903647 3053 webserver.cc:460] Webserver started at http://127.0.98.2:43663/ using document root <none> and password file <none>
I20260927 03:28:11.903932 3053 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:11.903997 3053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:11.904119 3053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:11.905421 3053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/data/instance:
uuid: "c5adbfadea1f449ca20f3e7944b0c912"
format_stamp: "Formatted at 2026-09-27 03:28:11 on dist-test-slave-j0hf"
server_key: "600b52f47aeb219682cde0cb0fdd9929"
server_key_iv: "af16bf324a3c8cf1f303a01ea2dc04a6"
server_key_version: "encryptionkey@0"
I20260927 03:28:11.905782 3053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c5adbfadea1f449ca20f3e7944b0c912"
format_stamp: "Formatted at 2026-09-27 03:28:11 on dist-test-slave-j0hf"
server_key: "600b52f47aeb219682cde0cb0fdd9929"
server_key_iv: "af16bf324a3c8cf1f303a01ea2dc04a6"
server_key_version: "encryptionkey@0"
I20260927 03:28:11.907094 3053 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:28:11.907950 3068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:11.908159 3053 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:28:11.908257 3053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "c5adbfadea1f449ca20f3e7944b0c912"
format_stamp: "Formatted at 2026-09-27 03:28:11 on dist-test-slave-j0hf"
server_key: "600b52f47aeb219682cde0cb0fdd9929"
server_key_iv: "af16bf324a3c8cf1f303a01ea2dc04a6"
server_key_version: "encryptionkey@0"
I20260927 03:28:11.908335 3053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:11.919891 3053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:11.922266 3053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:11.922415 3053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:11.922680 3053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:11.923008 3053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:11.923064 3053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:11.923139 3053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:11.923168 3053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:11.929219 3053 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43737
I20260927 03:28:11.929260 3181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43737 every 8 connection(s)
I20260927 03:28:11.929585 3053 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:28:11.931679 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 3053
I20260927 03:28:11.931777 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:28:11.931993 392 external_mini_cluster.cc:1428] Setting key 4a2178de50c10bbca8e7cae125f7b303
Sep 27 03:28:11 dist-test-slave-j0hf krb5kdc[2812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:11.940182 2897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:11.930365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34235 (local address 127.0.98.62:35403)
0927 03:28:11.930447 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:11.930450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:11.930651 (+ 201us) server_negotiation.cc:407] Connection header received
0927 03:28:11.931634 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:11.931636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:11.931651 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:11.931733 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:11.932644 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:11.933394 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:11.934479 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:11.934797 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:11.934876 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:11.937079 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:11.937081 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:11.937081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:11.937091 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:11.938704 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:11.939145 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:11.939147 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:11.939147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:11.939209 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:11.939522 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:11.939523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:11.939523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:11.939642 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:11.939716 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:11.939997 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 03:28:11.940087 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 03:28:11.940392 3184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:11.930316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35403 (local address 127.0.98.2:34235)
0927 03:28:11.930503 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:11.930539 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:11.931511 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:11.931909 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:11.931913 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:11.932240 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:11.932515 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:11.932521 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:11.933486 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:11.933487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:11.934395 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:11.934397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:11.934882 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:11.934883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:11.934992 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:11.935602 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:11.935611 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:11.936978 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:11.938810 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:11.938817 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:11.938820 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:11.939051 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:11.939286 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:11.939287 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:11.939287 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:11.939459 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:11.939706 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:11.939713 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:11.939860 (+ 147us) client_negotiation.cc:769] Sending connection context
0927 03:28:11.939964 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 03:28:11.940126 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":99,"threads_started":1}
I20260927 03:28:11.941022 3182 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35403
I20260927 03:28:11.941107 3182 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:11.941318 3182 heartbeater.cc:511] Master 127.0.98.62:35403 requested a full tablet report, sending...
I20260927 03:28:11.941802 2857 ts_manager.cc:194] Registered new tserver with Master: c5adbfadea1f449ca20f3e7944b0c912 (127.0.98.2:43737)
I20260927 03:28:11.942442 2857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34235
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:28:11.959266 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35403
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:12.040725 3187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:12.040948 3187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:12.040972 3187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:12.042074 3187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:12.042109 3187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:12.042135 3187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:28:12.043627 3187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:12.043823 3187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:12.043995 3187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:12.049307 3192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:12.049351 3195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:12.049307 3193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:12.049609 3187 server_base.cc:1034] running on GCE node
I20260927 03:28:12.049832 3187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:12.050033 3187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:12.051169 3187 hybrid_clock.cc:648] HybridClock initialized: now 1790479692051150 us; error 36 us; skew 500 ppm
Sep 27 03:28:12 dist-test-slave-j0hf krb5kdc[2812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:12.053755 3187 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:28:12.054311 3187 webserver.cc:460] Webserver started at http://127.0.98.3:33297/ using document root <none> and password file <none>
I20260927 03:28:12.054541 3187 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:12.054589 3187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:12.054740 3187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:12.055907 3187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/data/instance:
uuid: "c5a993b25a6a488baaef9538482532ed"
format_stamp: "Formatted at 2026-09-27 03:28:12 on dist-test-slave-j0hf"
server_key: "7d707f79a1819f5efd669ab28cfcd042"
server_key_iv: "5d645f45d9e50f8b1da218d85d8eb29d"
server_key_version: "encryptionkey@0"
I20260927 03:28:12.056298 3187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c5a993b25a6a488baaef9538482532ed"
format_stamp: "Formatted at 2026-09-27 03:28:12 on dist-test-slave-j0hf"
server_key: "7d707f79a1819f5efd669ab28cfcd042"
server_key_iv: "5d645f45d9e50f8b1da218d85d8eb29d"
server_key_version: "encryptionkey@0"
I20260927 03:28:12.057629 3187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:28:12.058353 3202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:12.058531 3187 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:12.058619 3187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "c5a993b25a6a488baaef9538482532ed"
format_stamp: "Formatted at 2026-09-27 03:28:12 on dist-test-slave-j0hf"
server_key: "7d707f79a1819f5efd669ab28cfcd042"
server_key_iv: "5d645f45d9e50f8b1da218d85d8eb29d"
server_key_version: "encryptionkey@0"
I20260927 03:28:12.058708 3187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:12.070591 3187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:12.072751 3187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:12.072901 3187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:12.073169 3187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:12.073513 3187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:12.073567 3187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:12.073609 3187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:12.073637 3187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:12.079823 3187 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35259
I20260927 03:28:12.079901 3315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35259 every 8 connection(s)
I20260927 03:28:12.080265 3187 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:28:12.084182 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 3187
I20260927 03:28:12.084318 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:28:12.084533 392 external_mini_cluster.cc:1428] Setting key 575a55538babb574d74cb098a6d6fa68
Sep 27 03:28:12 dist-test-slave-j0hf krb5kdc[2812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:12.090507 2897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:12.081089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49225 (local address 127.0.98.62:35403)
0927 03:28:12.081130 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:12.081132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:12.081441 (+ 309us) server_negotiation.cc:407] Connection header received
0927 03:28:12.082476 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:12.082477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:12.082490 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:12.082549 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:12.083225 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:12.083800 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:12.084995 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:12.085191 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:12.085282 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:12.087576 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:12.087577 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:12.087577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:12.087586 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:12.089068 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:12.089494 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:12.089495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:12.089495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:12.089557 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:12.089817 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:12.089818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:12.089819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:12.089940 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:12.090018 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:12.090319 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 03:28:12.090408 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:28:12.090719 3318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:12.081079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35403 (local address 127.0.98.3:49225)
0927 03:28:12.081258 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:12.081323 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:12.082350 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:12.082630 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:12.082635 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:12.082923 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:12.083119 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:12.083123 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:12.083895 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:12.083896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:12.084891 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:12.084894 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:12.085292 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:12.085293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:12.085421 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:12.085988 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:12.085996 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:12.087417 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:12.089170 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:12.089177 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:12.089180 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:12.089414 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:12.089625 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:12.089626 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:12.089626 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:12.089742 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:12.090027 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:12.090034 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:12.090178 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 03:28:12.090299 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 03:28:12.090467 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":86,"threads_started":1}
I20260927 03:28:12.091291 3316 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35403
I20260927 03:28:12.091392 3316 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:12.091604 3316 heartbeater.cc:511] Master 127.0.98.62:35403 requested a full tablet report, sending...
I20260927 03:28:12.092067 2857 ts_manager.cc:194] Registered new tserver with Master: c5a993b25a6a488baaef9538482532ed (127.0.98.3:35259)
I20260927 03:28:12.092545 2857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49225
I20260927 03:28:12.096288 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 03:28:12.107843 2897 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:12.106770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35456 (local address 127.0.98.62:35403)
0927 03:28:12.106887 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:12.106889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:12.106969 (+ 80us) server_negotiation.cc:407] Connection header received
0927 03:28:12.107032 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:12.107033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:12.107042 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:12.107093 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:12.107764 (+ 671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35456: BlockingRecv error: recv got EOF from 127.0.0.1:35456 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 03:28:12.109211 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 2919
I20260927 03:28:12.113588 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 3053
I20260927 03:28:12.118273 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 3187
I20260927 03:28:12.122591 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 2825
2026-09-27T03:28:12Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1850 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:28:12 dist-test-slave-j0hf krb5kdc[3328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:28:12 dist-test-slave-j0hf krb5kdc[3328](info): set up 2 sockets
Sep 27 03:28:12 dist-test-slave-j0hf krb5kdc[3328](info): commencing operation
krb5kdc: starting...
W20260927 03:28:13.191327 392 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:28:13 dist-test-slave-j0hf krb5kdc[3328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:28:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:28:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:28:13.314570 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46593
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:13.398032 3341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:13.398317 3341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:13.398351 3341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:13.400069 3341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:28:13.400115 3341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:13.400132 3341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:13.400148 3341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:28:13.400174 3341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:28:13.402271 3341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:13.402581 3341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:13.402805 3341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:13.407948 3349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:13.408003 3341 server_base.cc:1034] running on GCE node
W20260927 03:28:13.407927 3346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:13.407927 3347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:13.408461 3341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:13.408826 3341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:13.410008 3341 hybrid_clock.cc:648] HybridClock initialized: now 1790479693409981 us; error 51 us; skew 500 ppm
Sep 27 03:28:13 dist-test-slave-j0hf krb5kdc[3328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:13.412909 3341 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:28:13.413429 3341 webserver.cc:460] Webserver started at http://127.0.98.62:40397/ using document root <none> and password file <none>
I20260927 03:28:13.413693 3341 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:13.413789 3341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:13.413947 3341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:13.415088 3341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "14a978abfe3c4d37b58a549c7a16ac14"
format_stamp: "Formatted at 2026-09-27 03:28:13 on dist-test-slave-j0hf"
server_key: "7b8a5e3be57c0f65b08af969e3ef6a49"
server_key_iv: "90da68be2cae5c4220cb42f8fa62d17f"
server_key_version: "encryptionkey@0"
I20260927 03:28:13.415532 3341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "14a978abfe3c4d37b58a549c7a16ac14"
format_stamp: "Formatted at 2026-09-27 03:28:13 on dist-test-slave-j0hf"
server_key: "7b8a5e3be57c0f65b08af969e3ef6a49"
server_key_iv: "90da68be2cae5c4220cb42f8fa62d17f"
server_key_version: "encryptionkey@0"
I20260927 03:28:13.416939 3341 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:28:13.417742 3356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:13.417941 3341 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:28:13.418041 3341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "14a978abfe3c4d37b58a549c7a16ac14"
format_stamp: "Formatted at 2026-09-27 03:28:13 on dist-test-slave-j0hf"
server_key: "7b8a5e3be57c0f65b08af969e3ef6a49"
server_key_iv: "90da68be2cae5c4220cb42f8fa62d17f"
server_key_version: "encryptionkey@0"
I20260927 03:28:13.418125 3341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:13.425992 3341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:13.428524 3341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:13.428699 3341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:13.432922 3341 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46593
I20260927 03:28:13.432955 3408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46593 every 8 connection(s)
I20260927 03:28:13.433350 3341 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:28:13.434379 3409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:13.436828 3409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14a978abfe3c4d37b58a549c7a16ac14: Bootstrap starting.
I20260927 03:28:13.437458 3409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14a978abfe3c4d37b58a549c7a16ac14: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:13.437775 3409 log.cc:824] T 00000000000000000000000000000000 P 14a978abfe3c4d37b58a549c7a16ac14: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:13.438588 3409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14a978abfe3c4d37b58a549c7a16ac14: No bootstrap required, opened a new log
I20260927 03:28:13.440337 3409 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14a978abfe3c4d37b58a549c7a16ac14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a978abfe3c4d37b58a549c7a16ac14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46593 } }
I20260927 03:28:13.440488 3409 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14a978abfe3c4d37b58a549c7a16ac14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:13.440512 3409 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14a978abfe3c4d37b58a549c7a16ac14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14a978abfe3c4d37b58a549c7a16ac14, State: Initialized, Role: FOLLOWER
I20260927 03:28:13.440683 3409 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14a978abfe3c4d37b58a549c7a16ac14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a978abfe3c4d37b58a549c7a16ac14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46593 } }
I20260927 03:28:13.440760 3409 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14a978abfe3c4d37b58a549c7a16ac14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:28:13.440783 3409 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14a978abfe3c4d37b58a549c7a16ac14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:28:13.440851 3409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14a978abfe3c4d37b58a549c7a16ac14 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:13.441745 3409 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14a978abfe3c4d37b58a549c7a16ac14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a978abfe3c4d37b58a549c7a16ac14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46593 } }
I20260927 03:28:13.441836 3409 leader_election.cc:302] T 00000000000000000000000000000000 P 14a978abfe3c4d37b58a549c7a16ac14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14a978abfe3c4d37b58a549c7a16ac14; no voters:
I20260927 03:28:13.442070 3409 leader_election.cc:288] T 00000000000000000000000000000000 P 14a978abfe3c4d37b58a549c7a16ac14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:28:13.442112 3412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14a978abfe3c4d37b58a549c7a16ac14 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:13.442281 3412 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14a978abfe3c4d37b58a549c7a16ac14 [term 1 LEADER]: Becoming Leader. State: Replica: 14a978abfe3c4d37b58a549c7a16ac14, State: Running, Role: LEADER
I20260927 03:28:13.442353 3409 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14a978abfe3c4d37b58a549c7a16ac14 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:28:13.442370 3412 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14a978abfe3c4d37b58a549c7a16ac14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a978abfe3c4d37b58a549c7a16ac14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46593 } }
I20260927 03:28:13.442764 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 3341
I20260927 03:28:13.442801 3413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14a978abfe3c4d37b58a549c7a16ac14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14a978abfe3c4d37b58a549c7a16ac14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a978abfe3c4d37b58a549c7a16ac14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46593 } } }
I20260927 03:28:13.442885 3413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14a978abfe3c4d37b58a549c7a16ac14 [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:13.442898 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:28:13.443063 3414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14a978abfe3c4d37b58a549c7a16ac14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14a978abfe3c4d37b58a549c7a16ac14. Latest consensus state: current_term: 1 leader_uuid: "14a978abfe3c4d37b58a549c7a16ac14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a978abfe3c4d37b58a549c7a16ac14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46593 } } }
I20260927 03:28:13.443137 392 external_mini_cluster.cc:1428] Setting key 51a07411cf56254f9aa0d343c9c54063
I20260927 03:28:13.443114 3414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14a978abfe3c4d37b58a549c7a16ac14 [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:13.443408 3419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:28:13.443949 3419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:28:13.447871 3419 catalog_manager.cc:1347] Generated new cluster ID: 03a23fb76afe44638c29a4f098562d93
I20260927 03:28:13.447950 3419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:28:13.456326 3419 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:28:13.457044 3419 catalog_manager.cc:1504] Loading token signing keys...
Sep 27 03:28:13 dist-test-slave-j0hf krb5kdc[3328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:13.463284 3430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:13.443654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49298 (local address 127.0.98.62:46593)
0927 03:28:13.444801 (+ 1147us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:13.444808 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:13.444865 (+ 57us) server_negotiation.cc:407] Connection header received
0927 03:28:13.444989 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:13.445012 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:13.445093 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:13.445239 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:13.445931 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:13.446868 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:13.447735 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:13.455952 (+ 8217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:13.456146 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:13.459309 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:13.459314 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:13.459315 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:13.459328 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:13.461382 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:13.461827 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:13.461831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:13.461831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:13.461941 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:13.462170 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:13.462171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:13.462172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:13.462452 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:13.462614 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:13.462736 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 03:28:13.462881 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1101,"thread_start_us":97,"threads_started":1}
I20260927 03:28:13.464396 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:28:13.468874 3419 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14a978abfe3c4d37b58a549c7a16ac14: Generated new TSK 0
I20260927 03:28:13.469151 3419 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:28:13.481438 3430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:13.477535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49314 (local address 127.0.98.62:46593)
0927 03:28:13.477590 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:13.477592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:13.477685 (+ 93us) server_negotiation.cc:407] Connection header received
0927 03:28:13.477795 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:13.477796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:13.477814 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:13.477873 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:13.478305 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:13.478843 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:13.479693 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:13.479910 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:13.480002 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:13.480882 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:13.480884 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:13.480884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:28:13.481131 (+ 247us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:13.481150 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:28:13.481157 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:13.481233 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:13.481303 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 03:28:13.481338 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 03:28:13.484332 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 3341
2026-09-27T03:28:13Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1357 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:28:13 dist-test-slave-j0hf krb5kdc[3442](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:28:13 dist-test-slave-j0hf krb5kdc[3442](info): set up 2 sockets
Sep 27 03:28:13 dist-test-slave-j0hf krb5kdc[3442](info): commencing operation
krb5kdc: starting...
W20260927 03:28:14.535786 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260927 03:28:14.753005 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.217s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:28:14 dist-test-slave-j0hf krb5kdc[3442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:28:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:28:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:28:14.848459 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42775
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:14.930380 3455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:14.930608 3455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:14.930630 3455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:14.931898 3455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:28:14.931945 3455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:14.931960 3455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:14.931973 3455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:28:14.932036 3455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:28:14.933553 3455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:14.933849 3455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:14.934119 3455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:14.939055 3460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:14.939086 3463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:14.939056 3461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:14.939426 3455 server_base.cc:1034] running on GCE node
I20260927 03:28:14.939704 3455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:14.940043 3455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:14.941200 3455 hybrid_clock.cc:648] HybridClock initialized: now 1790479694941161 us; error 54 us; skew 500 ppm
Sep 27 03:28:14 dist-test-slave-j0hf krb5kdc[3442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:14.943917 3455 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:28:14.944525 3455 webserver.cc:460] Webserver started at http://127.0.98.62:32931/ using document root <none> and password file <none>
I20260927 03:28:14.944766 3455 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:14.944811 3455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:14.944967 3455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:14.946190 3455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "3d04abe6e162444eb79b83762e292ac4"
format_stamp: "Formatted at 2026-09-27 03:28:14 on dist-test-slave-j0hf"
server_key: "569a442592eaeefc0a1b6be84db91565"
server_key_iv: "573a69a4a2f2e6873b8d3808bb1d791c"
server_key_version: "encryptionkey@0"
I20260927 03:28:14.946578 3455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "3d04abe6e162444eb79b83762e292ac4"
format_stamp: "Formatted at 2026-09-27 03:28:14 on dist-test-slave-j0hf"
server_key: "569a442592eaeefc0a1b6be84db91565"
server_key_iv: "573a69a4a2f2e6873b8d3808bb1d791c"
server_key_version: "encryptionkey@0"
I20260927 03:28:14.948019 3455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:28:14.948802 3470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:14.948992 3455 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:28:14.949066 3455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "3d04abe6e162444eb79b83762e292ac4"
format_stamp: "Formatted at 2026-09-27 03:28:14 on dist-test-slave-j0hf"
server_key: "569a442592eaeefc0a1b6be84db91565"
server_key_iv: "573a69a4a2f2e6873b8d3808bb1d791c"
server_key_version: "encryptionkey@0"
I20260927 03:28:14.949152 3455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:14.961299 3455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:14.963910 3455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:14.964092 3455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:14.968683 3455 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42775
I20260927 03:28:14.968740 3522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42775 every 8 connection(s)
I20260927 03:28:14.969148 3455 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:28:14.970168 3523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:14.973026 3523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d04abe6e162444eb79b83762e292ac4: Bootstrap starting.
I20260927 03:28:14.973193 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 3455
I20260927 03:28:14.973284 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:28:14.973477 392 external_mini_cluster.cc:1428] Setting key 7cb06e0fb8c0c4d6203141c267933f4f
I20260927 03:28:14.973881 3523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d04abe6e162444eb79b83762e292ac4: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:14.974220 3523 log.cc:824] T 00000000000000000000000000000000 P 3d04abe6e162444eb79b83762e292ac4: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:14.974898 3523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d04abe6e162444eb79b83762e292ac4: No bootstrap required, opened a new log
I20260927 03:28:14.976866 3523 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d04abe6e162444eb79b83762e292ac4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d04abe6e162444eb79b83762e292ac4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42775 } }
I20260927 03:28:14.977005 3523 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d04abe6e162444eb79b83762e292ac4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:14.977030 3523 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d04abe6e162444eb79b83762e292ac4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d04abe6e162444eb79b83762e292ac4, State: Initialized, Role: FOLLOWER
I20260927 03:28:14.977185 3523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d04abe6e162444eb79b83762e292ac4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d04abe6e162444eb79b83762e292ac4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42775 } }
I20260927 03:28:14.977260 3523 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d04abe6e162444eb79b83762e292ac4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:28:14.977283 3523 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d04abe6e162444eb79b83762e292ac4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:28:14.977352 3523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d04abe6e162444eb79b83762e292ac4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:14.977921 3523 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d04abe6e162444eb79b83762e292ac4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d04abe6e162444eb79b83762e292ac4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42775 } }
I20260927 03:28:14.978044 3523 leader_election.cc:302] T 00000000000000000000000000000000 P 3d04abe6e162444eb79b83762e292ac4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d04abe6e162444eb79b83762e292ac4; no voters:
I20260927 03:28:14.978250 3523 leader_election.cc:288] T 00000000000000000000000000000000 P 3d04abe6e162444eb79b83762e292ac4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:28:14.978271 3528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d04abe6e162444eb79b83762e292ac4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:14.978392 3528 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d04abe6e162444eb79b83762e292ac4 [term 1 LEADER]: Becoming Leader. State: Replica: 3d04abe6e162444eb79b83762e292ac4, State: Running, Role: LEADER
I20260927 03:28:14.978482 3528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d04abe6e162444eb79b83762e292ac4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d04abe6e162444eb79b83762e292ac4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42775 } }
I20260927 03:28:14.978536 3523 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d04abe6e162444eb79b83762e292ac4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:28:14.978789 3528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d04abe6e162444eb79b83762e292ac4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d04abe6e162444eb79b83762e292ac4. Latest consensus state: current_term: 1 leader_uuid: "3d04abe6e162444eb79b83762e292ac4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d04abe6e162444eb79b83762e292ac4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42775 } } }
I20260927 03:28:14.978853 3528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d04abe6e162444eb79b83762e292ac4 [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:14.978978 3528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d04abe6e162444eb79b83762e292ac4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d04abe6e162444eb79b83762e292ac4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d04abe6e162444eb79b83762e292ac4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42775 } } }
I20260927 03:28:14.979051 3528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d04abe6e162444eb79b83762e292ac4 [sys.catalog]: This master's current role is: LEADER
W20260927 03:28:14.980408 3544 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3d04abe6e162444eb79b83762e292ac4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:28:14.980468 3544 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:28:14.980511 3532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:28:14.980661 3532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 03:28:14 dist-test-slave-j0hf krb5kdc[3442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:14.982247 3532 catalog_manager.cc:1347] Generated new cluster ID: eb12e1439ccb4869b52983a3c1be8465
I20260927 03:28:14.982300 3532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:28:14.986073 3526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:14.974061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58252 (local address 127.0.98.62:42775)
0927 03:28:14.974280 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:14.974291 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:14.974329 (+ 38us) server_negotiation.cc:407] Connection header received
0927 03:28:14.974896 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:14.974904 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:14.974989 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:14.975172 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:14.975883 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:14.976743 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:14.977741 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:14.978262 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:14.978419 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:14.981167 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:14.981171 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:14.981171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:14.981183 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:14.984379 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:14.984762 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:14.984766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:14.984766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:14.984890 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:14.985100 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:14.985101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:14.985102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:14.985419 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:14.985587 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:14.985671 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 03:28:14.985814 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":85,"threads_started":1}
I20260927 03:28:14.994724 3532 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:28:14.995236 3532 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:28:15.002420 3532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d04abe6e162444eb79b83762e292ac4: Generated new TSK 0
I20260927 03:28:15.002611 3532 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:28:15.030202 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42775
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:15.113380 3549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:15.113593 3549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:15.113617 3549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:15.114822 3549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:15.114869 3549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:15.114897 3549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:28:15.116436 3549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:15.116693 3549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:15.116941 3549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:15.122088 3555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:15.122088 3557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:15.122112 3554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:15.122355 3549 server_base.cc:1034] running on GCE node
I20260927 03:28:15.122542 3549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:15.122741 3549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:15.123881 3549 hybrid_clock.cc:648] HybridClock initialized: now 1790479695123867 us; error 29 us; skew 500 ppm
Sep 27 03:28:15 dist-test-slave-j0hf krb5kdc[3442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:15.126650 3549 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:28:15.127130 3549 webserver.cc:460] Webserver started at http://127.0.98.1:34527/ using document root <none> and password file <none>
I20260927 03:28:15.127357 3549 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:15.127424 3549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:15.127594 3549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:15.128670 3549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/data/instance:
uuid: "ea09242f1bcb4921b47570c4a22fbdce"
format_stamp: "Formatted at 2026-09-27 03:28:15 on dist-test-slave-j0hf"
server_key: "adccd9bfb6d8c79347ff64b4b97e2868"
server_key_iv: "77c0a5fa74fe36c9a2ee335dac17b3a2"
server_key_version: "encryptionkey@0"
I20260927 03:28:15.129046 3549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ea09242f1bcb4921b47570c4a22fbdce"
format_stamp: "Formatted at 2026-09-27 03:28:15 on dist-test-slave-j0hf"
server_key: "adccd9bfb6d8c79347ff64b4b97e2868"
server_key_iv: "77c0a5fa74fe36c9a2ee335dac17b3a2"
server_key_version: "encryptionkey@0"
I20260927 03:28:15.130369 3549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:28:15.131145 3564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:15.131319 3549 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:28:15.131417 3549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "ea09242f1bcb4921b47570c4a22fbdce"
format_stamp: "Formatted at 2026-09-27 03:28:15 on dist-test-slave-j0hf"
server_key: "adccd9bfb6d8c79347ff64b4b97e2868"
server_key_iv: "77c0a5fa74fe36c9a2ee335dac17b3a2"
server_key_version: "encryptionkey@0"
I20260927 03:28:15.131528 3549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:15.155048 3549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:15.157233 3549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:15.157413 3549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:15.157668 3549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:15.158015 3549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:15.158072 3549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:15.158118 3549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:15.158141 3549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:15.164202 3549 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34723
I20260927 03:28:15.164291 3677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34723 every 8 connection(s)
I20260927 03:28:15.164647 3549 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:28:15.165370 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 3549
I20260927 03:28:15.165485 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:28:15.165732 392 external_mini_cluster.cc:1428] Setting key 87e6f3959cf2edb96dd54e9e93540242
Sep 27 03:28:15 dist-test-slave-j0hf krb5kdc[3442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 03:28:15.176010 3526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:15.165491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47179 (local address 127.0.98.62:42775)
0927 03:28:15.165559 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:15.165561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:15.165813 (+ 252us) server_negotiation.cc:407] Connection header received
0927 03:28:15.166373 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:15.166374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:15.166385 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:15.166426 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:15.167479 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:15.167997 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:15.169497 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:15.169707 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:15.169784 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:15.172691 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:15.172694 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:15.172694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:15.172702 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:15.174421 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:15.174869 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:15.174870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:15.174871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:15.174951 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:15.175245 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:15.175246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:15.175246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:15.175359 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:15.175410 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:15.175812 (+ 402us) server_negotiation.cc:299] Negotiation successful
0927 03:28:15.175936 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:28:15.176232 3680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:15.165456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42775 (local address 127.0.98.1:47179)
0927 03:28:15.165652 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:15.165697 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:15.166262 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:15.166531 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:15.166535 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:15.167128 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:15.167332 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:15.167337 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:15.168111 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:15.168113 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:15.169354 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:15.169358 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:15.169817 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:15.169818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:15.170012 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:15.170611 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:15.170622 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:15.172559 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:15.174535 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:15.174542 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:15.174544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:15.174796 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:15.175046 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:15.175047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:15.175048 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:15.175193 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:15.175509 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:15.175522 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:15.175671 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 03:28:15.175779 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 03:28:15.175965 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":90,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 03:28:15.176769 3678 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42775
I20260927 03:28:15.176901 3678 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:15.177139 3678 heartbeater.cc:511] Master 127.0.98.62:42775 requested a full tablet report, sending...
I20260927 03:28:15.177722 3487 ts_manager.cc:194] Registered new tserver with Master: ea09242f1bcb4921b47570c4a22fbdce (127.0.98.1:34723)
I20260927 03:28:15.178706 3487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:28:15.194223 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42775
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:15.276541 3683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:15.276765 3683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:15.276791 3683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:15.278136 3683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:15.278182 3683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:15.278213 3683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:28:15.280082 3683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:15.280372 3683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:15.280627 3683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:15.286053 3691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:15.286037 3689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:15.286137 3683 server_base.cc:1034] running on GCE node
W20260927 03:28:15.286022 3688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:15.286496 3683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:15.286765 3683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:15.287943 3683 hybrid_clock.cc:648] HybridClock initialized: now 1790479695287939 us; error 38 us; skew 500 ppm
Sep 27 03:28:15 dist-test-slave-j0hf krb5kdc[3442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:15.290750 3683 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:28:15.291208 3683 webserver.cc:460] Webserver started at http://127.0.98.2:36483/ using document root <none> and password file <none>
I20260927 03:28:15.291502 3683 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:15.291574 3683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:15.291740 3683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:15.292825 3683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/data/instance:
uuid: "23a107fab5a14509a2995a833de62de2"
format_stamp: "Formatted at 2026-09-27 03:28:15 on dist-test-slave-j0hf"
server_key: "c7558468e8091b746ec078971dca350d"
server_key_iv: "00aa0ed16702fbd6503ed11382cfe27f"
server_key_version: "encryptionkey@0"
I20260927 03:28:15.293208 3683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/wal/instance:
uuid: "23a107fab5a14509a2995a833de62de2"
format_stamp: "Formatted at 2026-09-27 03:28:15 on dist-test-slave-j0hf"
server_key: "c7558468e8091b746ec078971dca350d"
server_key_iv: "00aa0ed16702fbd6503ed11382cfe27f"
server_key_version: "encryptionkey@0"
I20260927 03:28:15.294610 3683 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:28:15.295404 3698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:15.295672 3683 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:28:15.295789 3683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "23a107fab5a14509a2995a833de62de2"
format_stamp: "Formatted at 2026-09-27 03:28:15 on dist-test-slave-j0hf"
server_key: "c7558468e8091b746ec078971dca350d"
server_key_iv: "00aa0ed16702fbd6503ed11382cfe27f"
server_key_version: "encryptionkey@0"
I20260927 03:28:15.295884 3683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:15.315868 3683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:15.318380 3683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:15.318553 3683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:15.318811 3683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:15.319134 3683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:15.319195 3683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:15.319242 3683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:15.319264 3683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:15.325349 3683 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38969
I20260927 03:28:15.325433 3811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38969 every 8 connection(s)
I20260927 03:28:15.325771 3683 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:28:15.329567 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 3683
I20260927 03:28:15.329684 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:28:15.329865 392 external_mini_cluster.cc:1428] Setting key ed7fae42c223315e44ea52bd37e01f27
Sep 27 03:28:15 dist-test-slave-j0hf krb5kdc[3442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:15.337217 3526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:15.326557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52659 (local address 127.0.98.62:42775)
0927 03:28:15.326629 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:15.326631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:15.326804 (+ 173us) server_negotiation.cc:407] Connection header received
0927 03:28:15.327298 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:15.327299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:15.327309 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:15.327375 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:15.328409 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:15.328911 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:15.330249 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:15.330549 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:15.330633 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:15.333667 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:15.333669 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:15.333669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:15.333678 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:15.335639 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:15.336173 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:15.336173 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:15.336174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:15.336232 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:15.336546 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:15.336547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:15.336548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:15.336671 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:15.336735 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:15.337079 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 03:28:15.337150 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 03:28:15.337575 3814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:15.326527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42775 (local address 127.0.98.2:52659)
0927 03:28:15.326671 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:15.326721 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:15.327210 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:15.327505 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:15.327509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:15.328100 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:15.328301 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:15.328305 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:15.328995 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:15.328996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:15.330140 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:15.330143 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:15.330643 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:15.330660 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:15.330819 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:15.331415 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:15.331459 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:15.333558 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:15.335742 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:15.335750 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:15.335753 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:15.336092 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:15.336307 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:15.336307 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:15.336308 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:15.336466 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:15.336747 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:15.336753 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:15.336930 (+ 177us) client_negotiation.cc:769] Sending connection context
0927 03:28:15.337055 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 03:28:15.337250 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":77,"threads_started":1}
I20260927 03:28:15.338114 3812 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42775
I20260927 03:28:15.338240 3812 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:15.338430 3812 heartbeater.cc:511] Master 127.0.98.62:42775 requested a full tablet report, sending...
I20260927 03:28:15.338833 3487 ts_manager.cc:194] Registered new tserver with Master: 23a107fab5a14509a2995a833de62de2 (127.0.98.2:38969)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 03:28:15.339265 3487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52659
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:28:15.357779 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42775
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:15.443713 3817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:15.443938 3817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:15.443969 3817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:15.445197 3817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:15.445237 3817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:15.445264 3817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:28:15.446719 3817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:15.446919 3817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:15.447100 3817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:15.452250 3823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:15.452247 3825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:15.452229 3822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:15.452534 3817 server_base.cc:1034] running on GCE node
I20260927 03:28:15.452780 3817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:15.453033 3817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:15.454190 3817 hybrid_clock.cc:648] HybridClock initialized: now 1790479695454164 us; error 32 us; skew 500 ppm
Sep 27 03:28:15 dist-test-slave-j0hf krb5kdc[3442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:15.456724 3817 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:28:15.457219 3817 webserver.cc:460] Webserver started at http://127.0.98.3:44365/ using document root <none> and password file <none>
I20260927 03:28:15.457463 3817 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:15.457531 3817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:15.457686 3817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:15.458743 3817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/data/instance:
uuid: "008c2e9c76014d0c907f88fb38d1f6f7"
format_stamp: "Formatted at 2026-09-27 03:28:15 on dist-test-slave-j0hf"
server_key: "dc1121471b9269c204535b954e5e2402"
server_key_iv: "95575cc45fab07f801b21bb38cbb736c"
server_key_version: "encryptionkey@0"
I20260927 03:28:15.459107 3817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/wal/instance:
uuid: "008c2e9c76014d0c907f88fb38d1f6f7"
format_stamp: "Formatted at 2026-09-27 03:28:15 on dist-test-slave-j0hf"
server_key: "dc1121471b9269c204535b954e5e2402"
server_key_iv: "95575cc45fab07f801b21bb38cbb736c"
server_key_version: "encryptionkey@0"
I20260927 03:28:15.460621 3817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:28:15.461356 3832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:15.461522 3817 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:15.461627 3817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "008c2e9c76014d0c907f88fb38d1f6f7"
format_stamp: "Formatted at 2026-09-27 03:28:15 on dist-test-slave-j0hf"
server_key: "dc1121471b9269c204535b954e5e2402"
server_key_iv: "95575cc45fab07f801b21bb38cbb736c"
server_key_version: "encryptionkey@0"
I20260927 03:28:15.461697 3817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:15.506160 3817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:15.508550 3817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:15.508730 3817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:15.508983 3817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:15.509318 3817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:15.509373 3817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:15.509419 3817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:15.509451 3817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:15.515518 3817 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35049
I20260927 03:28:15.515578 3945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35049 every 8 connection(s)
I20260927 03:28:15.515936 3817 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:28:15.516597 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 3817
I20260927 03:28:15.516698 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790479675865103-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:28:15.516922 392 external_mini_cluster.cc:1428] Setting key f63b0b6d31b843e82e7971bf64740e28
Sep 27 03:28:15 dist-test-slave-j0hf krb5kdc[3442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:15.525188 3526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:15.516712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48835 (local address 127.0.98.62:42775)
0927 03:28:15.516785 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:15.516787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:15.517040 (+ 253us) server_negotiation.cc:407] Connection header received
0927 03:28:15.517798 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:15.517799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:15.517808 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:15.517850 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:15.518826 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:15.519318 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:15.520377 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:15.520558 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:15.520612 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:15.522788 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:15.522789 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:15.522789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:15.522796 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:15.524049 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:15.524400 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:15.524401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:15.524401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:15.524447 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:15.524669 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:15.524670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:15.524670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:15.524776 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:15.524825 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:15.525063 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 03:28:15.525121 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:28:15.525420 3948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:15.516695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42775 (local address 127.0.98.3:48835)
0927 03:28:15.516881 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:15.516947 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:15.517687 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:15.517907 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:15.517910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:15.518494 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:15.518749 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:15.518753 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:15.519409 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:15.519410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:15.520301 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:15.520303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:15.520634 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:15.520635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:15.520748 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:15.521317 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:15.521326 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:15.522699 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:15.524121 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:15.524127 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:15.524129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:15.524336 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:15.524514 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:15.524514 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:15.524514 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:15.524615 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:15.524830 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:15.524834 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:15.524945 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 03:28:15.525038 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 03:28:15.525155 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":105,"threads_started":1}
I20260927 03:28:15.525950 3946 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42775
I20260927 03:28:15.526087 3946 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:15.526270 3946 heartbeater.cc:511] Master 127.0.98.62:42775 requested a full tablet report, sending...
I20260927 03:28:15.526654 3487 ts_manager.cc:194] Registered new tserver with Master: 008c2e9c76014d0c907f88fb38d1f6f7 (127.0.98.3:35049)
I20260927 03:28:15.527030 3487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48835
I20260927 03:28:15.528535 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 03:28:15 dist-test-slave-j0hf krb5kdc[3442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479695, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 03:28:15 dist-test-slave-j0hf krb5kdc[3442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479695, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:15.550834 3526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:15.544046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58268 (local address 127.0.98.62:42775)
0927 03:28:15.544124 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:15.544126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:15.544266 (+ 140us) server_negotiation.cc:407] Connection header received
0927 03:28:15.544315 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:15.544316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:15.544326 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:15.544382 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:15.544932 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:15.545462 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:15.546164 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:15.546337 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:15.546386 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:15.548660 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:15.548662 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:15.548662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:15.548669 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:15.549894 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:15.550209 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:15.550210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:15.550210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:15.550257 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:15.550412 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:15.550412 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:15.550413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:15.550547 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:15.550621 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:15.550707 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 03:28:15.550766 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 03:28:15.551069 3487 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58268
I20260927 03:28:15.551522 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 3549
I20260927 03:28:15.556540 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 3683
I20260927 03:28:15.560691 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 3817
I20260927 03:28:15.565112 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 3455
2026-09-27T03:28:15Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2083 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:28:15 dist-test-slave-j0hf krb5kdc[3958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:28:15 dist-test-slave-j0hf krb5kdc[3958](info): set up 2 sockets
Sep 27 03:28:15 dist-test-slave-j0hf krb5kdc[3958](info): commencing operation
krb5kdc: starting...
W20260927 03:28:16.607360 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:28:16 dist-test-slave-j0hf krb5kdc[3958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:28:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:28:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:28:16.726002 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38523
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:16.806074 3971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:16.806336 3971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:16.806373 3971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:16.807570 3971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:28:16.807649 3971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:16.807679 3971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:16.807710 3971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:28:16.807734 3971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:28:16.809180 3971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:16.809458 3971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:16.809695 3971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:16.814491 3976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:16.814483 3979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:16.814934 3977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:16.815186 3971 server_base.cc:1034] running on GCE node
I20260927 03:28:16.815383 3971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:16.815729 3971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:16.816872 3971 hybrid_clock.cc:648] HybridClock initialized: now 1790479696816851 us; error 37 us; skew 500 ppm
Sep 27 03:28:16 dist-test-slave-j0hf krb5kdc[3958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:16.824110 3971 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:28:16.824565 3971 webserver.cc:460] Webserver started at http://127.0.98.62:46271/ using document root <none> and password file <none>
I20260927 03:28:16.824826 3971 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:16.824895 3971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:16.825044 3971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:16.826141 3971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "f470536551704094969a7c18b9a4ad68"
format_stamp: "Formatted at 2026-09-27 03:28:16 on dist-test-slave-j0hf"
server_key: "22ec7fdd04acb8866b11358d541ced54"
server_key_iv: "66cf2f870f9caa0f5e734340f32c2602"
server_key_version: "encryptionkey@0"
I20260927 03:28:16.826511 3971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "f470536551704094969a7c18b9a4ad68"
format_stamp: "Formatted at 2026-09-27 03:28:16 on dist-test-slave-j0hf"
server_key: "22ec7fdd04acb8866b11358d541ced54"
server_key_iv: "66cf2f870f9caa0f5e734340f32c2602"
server_key_version: "encryptionkey@0"
I20260927 03:28:16.827894 3971 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:28:16.828591 3986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:16.828763 3971 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:28:16.828855 3971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "f470536551704094969a7c18b9a4ad68"
format_stamp: "Formatted at 2026-09-27 03:28:16 on dist-test-slave-j0hf"
server_key: "22ec7fdd04acb8866b11358d541ced54"
server_key_iv: "66cf2f870f9caa0f5e734340f32c2602"
server_key_version: "encryptionkey@0"
I20260927 03:28:16.828936 3971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:16.844540 3971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:16.846940 3971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:16.847103 3971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:16.850881 3971 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38523
I20260927 03:28:16.850915 4038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38523 every 8 connection(s)
I20260927 03:28:16.851260 3971 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:28:16.852188 4039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:16.854472 4039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f470536551704094969a7c18b9a4ad68: Bootstrap starting.
I20260927 03:28:16.855047 4039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f470536551704094969a7c18b9a4ad68: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:16.855346 4039 log.cc:824] T 00000000000000000000000000000000 P f470536551704094969a7c18b9a4ad68: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:16.856037 4039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f470536551704094969a7c18b9a4ad68: No bootstrap required, opened a new log
I20260927 03:28:16.857635 4039 raft_consensus.cc:348] T 00000000000000000000000000000000 P f470536551704094969a7c18b9a4ad68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f470536551704094969a7c18b9a4ad68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38523 } }
I20260927 03:28:16.857774 4039 raft_consensus.cc:374] T 00000000000000000000000000000000 P f470536551704094969a7c18b9a4ad68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:16.857795 4039 raft_consensus.cc:729] T 00000000000000000000000000000000 P f470536551704094969a7c18b9a4ad68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f470536551704094969a7c18b9a4ad68, State: Initialized, Role: FOLLOWER
I20260927 03:28:16.857937 4039 consensus_queue.cc:260] T 00000000000000000000000000000000 P f470536551704094969a7c18b9a4ad68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f470536551704094969a7c18b9a4ad68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38523 } }
I20260927 03:28:16.858013 4039 raft_consensus.cc:388] T 00000000000000000000000000000000 P f470536551704094969a7c18b9a4ad68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:28:16.858036 4039 raft_consensus.cc:482] T 00000000000000000000000000000000 P f470536551704094969a7c18b9a4ad68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:28:16.858086 4039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f470536551704094969a7c18b9a4ad68 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:16.858603 4039 raft_consensus.cc:504] T 00000000000000000000000000000000 P f470536551704094969a7c18b9a4ad68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f470536551704094969a7c18b9a4ad68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38523 } }
I20260927 03:28:16.858690 4039 leader_election.cc:302] T 00000000000000000000000000000000 P f470536551704094969a7c18b9a4ad68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f470536551704094969a7c18b9a4ad68; no voters:
I20260927 03:28:16.858897 4039 leader_election.cc:288] T 00000000000000000000000000000000 P f470536551704094969a7c18b9a4ad68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:28:16.858969 4042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f470536551704094969a7c18b9a4ad68 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:16.859153 4039 sys_catalog.cc:564] T 00000000000000000000000000000000 P f470536551704094969a7c18b9a4ad68 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:28:16.859382 4042 raft_consensus.cc:686] T 00000000000000000000000000000000 P f470536551704094969a7c18b9a4ad68 [term 1 LEADER]: Becoming Leader. State: Replica: f470536551704094969a7c18b9a4ad68, State: Running, Role: LEADER
I20260927 03:28:16.859493 4042 consensus_queue.cc:237] T 00000000000000000000000000000000 P f470536551704094969a7c18b9a4ad68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f470536551704094969a7c18b9a4ad68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38523 } }
I20260927 03:28:16.859876 4044 sys_catalog.cc:455] T 00000000000000000000000000000000 P f470536551704094969a7c18b9a4ad68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f470536551704094969a7c18b9a4ad68. Latest consensus state: current_term: 1 leader_uuid: "f470536551704094969a7c18b9a4ad68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f470536551704094969a7c18b9a4ad68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38523 } } }
I20260927 03:28:16.859884 4043 sys_catalog.cc:455] T 00000000000000000000000000000000 P f470536551704094969a7c18b9a4ad68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f470536551704094969a7c18b9a4ad68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f470536551704094969a7c18b9a4ad68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38523 } } }
I20260927 03:28:16.860021 4044 sys_catalog.cc:458] T 00000000000000000000000000000000 P f470536551704094969a7c18b9a4ad68 [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:16.860059 4043 sys_catalog.cc:458] T 00000000000000000000000000000000 P f470536551704094969a7c18b9a4ad68 [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:16.860440 4051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:28:16.860984 4051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:28:16.860980 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 3971
I20260927 03:28:16.861081 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:28:16.861286 392 external_mini_cluster.cc:1428] Setting key 08c655f72e8692ac413b1fa77e36c77e
I20260927 03:28:16.862815 4051 catalog_manager.cc:1347] Generated new cluster ID: 53e808af2082401aa012d665aef29c45
I20260927 03:28:16.862869 4051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:28:16 dist-test-slave-j0hf krb5kdc[3958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:16.871409 4061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:16.861961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55418 (local address 127.0.98.62:38523)
0927 03:28:16.862132 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:16.862148 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:16.862182 (+ 34us) server_negotiation.cc:407] Connection header received
0927 03:28:16.862298 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:16.862303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:16.862350 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:16.862500 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:16.863286 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:16.864309 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:16.865084 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:16.865522 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:16.865655 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:16.868085 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:16.868091 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:16.868092 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:16.868108 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:16.869834 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:16.870211 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:16.870222 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:16.870222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:16.870309 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:16.870517 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:16.870518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:16.870518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:16.870763 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:16.870936 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:16.871035 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 03:28:16.871189 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":93,"threads_started":1}
I20260927 03:28:16.872888 4051 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:28:16.873577 4051 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:28:16.877873 4051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f470536551704094969a7c18b9a4ad68: Generated new TSK 0
I20260927 03:28:16.878041 4051 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:28:16.908421 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38523
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:16.999902 4065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:17.000125 4065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:17.000150 4065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:17.001365 4065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:17.001410 4065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:17.001439 4065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:28:17.002954 4065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:17.003214 4065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:17.003533 4065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:17.008565 4070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:17.008569 4073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:17.008810 4071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:17.009027 4065 server_base.cc:1034] running on GCE node
I20260927 03:28:17.009224 4065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:17.009475 4065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:17.010656 4065 hybrid_clock.cc:648] HybridClock initialized: now 1790479697010638 us; error 32 us; skew 500 ppm
Sep 27 03:28:17 dist-test-slave-j0hf krb5kdc[3958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:17.013572 4065 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:28:17.014088 4065 webserver.cc:460] Webserver started at http://127.0.98.1:42057/ using document root <none> and password file <none>
I20260927 03:28:17.014317 4065 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:17.014382 4065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:17.014544 4065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:17.015688 4065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-0/data/instance:
uuid: "123a52f3b4b74dd39cf856b3b8630698"
format_stamp: "Formatted at 2026-09-27 03:28:17 on dist-test-slave-j0hf"
server_key: "01fcb71dc76fd9c09a4fb6909d132676"
server_key_iv: "e3009316402bc6a374625cbc5366580d"
server_key_version: "encryptionkey@0"
I20260927 03:28:17.016045 4065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-0/wal/instance:
uuid: "123a52f3b4b74dd39cf856b3b8630698"
format_stamp: "Formatted at 2026-09-27 03:28:17 on dist-test-slave-j0hf"
server_key: "01fcb71dc76fd9c09a4fb6909d132676"
server_key_iv: "e3009316402bc6a374625cbc5366580d"
server_key_version: "encryptionkey@0"
I20260927 03:28:17.017354 4065 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:28:17.018074 4080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:17.018244 4065 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:17.018342 4065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "123a52f3b4b74dd39cf856b3b8630698"
format_stamp: "Formatted at 2026-09-27 03:28:17 on dist-test-slave-j0hf"
server_key: "01fcb71dc76fd9c09a4fb6909d132676"
server_key_iv: "e3009316402bc6a374625cbc5366580d"
server_key_version: "encryptionkey@0"
I20260927 03:28:17.018421 4065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:17.028960 4065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:17.030822 4065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:17.031018 4065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:17.031266 4065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:17.031642 4065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:17.031697 4065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:17.031731 4065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:17.031759 4065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:17.037452 4065 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44397
I20260927 03:28:17.037528 4193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44397 every 8 connection(s)
I20260927 03:28:17.037860 4065 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:28:17 dist-test-slave-j0hf krb5kdc[3958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:17.044184 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 4065
I20260927 03:28:17.044299 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:28:17.044517 392 external_mini_cluster.cc:1428] Setting key 2bd69d37ed45f3eab0659cbab7390c5c
I20260927 03:28:17.047374 4061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:17.038594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35741 (local address 127.0.98.62:38523)
0927 03:28:17.038646 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:17.038648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:17.038846 (+ 198us) server_negotiation.cc:407] Connection header received
0927 03:28:17.039833 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:17.039834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:17.039848 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:17.039896 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:17.040455 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:17.040945 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:17.041920 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:17.042110 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:17.042187 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:17.044161 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:17.044163 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:17.044163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:17.044176 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:17.046024 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:17.046386 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:17.046387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:17.046388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:17.046450 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:17.046775 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:17.046776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:17.046776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:17.046877 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:17.046931 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:17.047245 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 03:28:17.047313 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 03:28:17.047693 4196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:17.038556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38523 (local address 127.0.98.1:35741)
0927 03:28:17.038723 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:17.038757 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:17.039739 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:17.039960 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:17.039964 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:17.040202 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:17.040385 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:17.040389 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:17.041029 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:17.041029 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:17.041847 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:17.041849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:17.042194 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:17.042195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:17.042291 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:17.042704 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:17.042711 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:17.044064 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:17.046110 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:17.046115 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:17.046117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:17.046319 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:17.046556 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:17.046557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:17.046557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:17.046704 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:17.046942 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:17.046948 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:17.047110 (+ 162us) client_negotiation.cc:769] Sending connection context
0927 03:28:17.047227 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 03:28:17.047398 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":101,"threads_started":1}
I20260927 03:28:17.048130 4194 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38523
I20260927 03:28:17.048216 4194 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:17.048405 4194 heartbeater.cc:511] Master 127.0.98.62:38523 requested a full tablet report, sending...
I20260927 03:28:17.048931 4003 ts_manager.cc:194] Registered new tserver with Master: 123a52f3b4b74dd39cf856b3b8630698 (127.0.98.1:44397)
I20260927 03:28:17.049543 4003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35741
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:28:17.069954 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38523
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:17.153617 4199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:17.153847 4199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:17.153870 4199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:17.154969 4199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:17.155009 4199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:17.155036 4199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:28:17.156495 4199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:17.156695 4199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:17.156884 4199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:17.162223 4205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:17.162185 4207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:17.162185 4204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:17.162258 4199 server_base.cc:1034] running on GCE node
I20260927 03:28:17.162603 4199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:17.162875 4199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:17.164018 4199 hybrid_clock.cc:648] HybridClock initialized: now 1790479697163998 us; error 45 us; skew 500 ppm
Sep 27 03:28:17 dist-test-slave-j0hf krb5kdc[3958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:17.166546 4199 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:28:17.167043 4199 webserver.cc:460] Webserver started at http://127.0.98.2:41013/ using document root <none> and password file <none>
I20260927 03:28:17.167299 4199 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:17.167346 4199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:17.167539 4199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:17.168637 4199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-1/data/instance:
uuid: "020a5e68f2084170aa1051a2dfbefedc"
format_stamp: "Formatted at 2026-09-27 03:28:17 on dist-test-slave-j0hf"
server_key: "d142313c214b7ba1d70285530955372f"
server_key_iv: "8e937545667765f4e11491c9464dddaa"
server_key_version: "encryptionkey@0"
I20260927 03:28:17.168989 4199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-1/wal/instance:
uuid: "020a5e68f2084170aa1051a2dfbefedc"
format_stamp: "Formatted at 2026-09-27 03:28:17 on dist-test-slave-j0hf"
server_key: "d142313c214b7ba1d70285530955372f"
server_key_iv: "8e937545667765f4e11491c9464dddaa"
server_key_version: "encryptionkey@0"
I20260927 03:28:17.170344 4199 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:28:17.171079 4214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:17.171276 4199 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:17.171367 4199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "020a5e68f2084170aa1051a2dfbefedc"
format_stamp: "Formatted at 2026-09-27 03:28:17 on dist-test-slave-j0hf"
server_key: "d142313c214b7ba1d70285530955372f"
server_key_iv: "8e937545667765f4e11491c9464dddaa"
server_key_version: "encryptionkey@0"
I20260927 03:28:17.171483 4199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:17.184126 4199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:17.186080 4199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:17.186246 4199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:17.186479 4199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:17.186796 4199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:17.186851 4199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:17.186887 4199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:17.186913 4199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:17.192842 4199 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35273
I20260927 03:28:17.192881 4327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35273 every 8 connection(s)
I20260927 03:28:17.193240 4199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:28:17.194834 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 4199
I20260927 03:28:17.194926 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:28:17.195127 392 external_mini_cluster.cc:1428] Setting key fb681b160b61518bfd28af79237f1d05
Sep 27 03:28:17 dist-test-slave-j0hf krb5kdc[3958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 03:28:17.204247 4061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:17.194075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59875 (local address 127.0.98.62:38523)
0927 03:28:17.194148 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:17.194167 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:17.194359 (+ 192us) server_negotiation.cc:407] Connection header received
0927 03:28:17.195683 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:17.195684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:17.195693 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:17.195747 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:17.196486 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:17.196975 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:17.198044 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:17.198369 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:17.198442 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:17.201221 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:17.201223 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:17.201223 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:17.201231 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:17.202680 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:17.203182 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:17.203182 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:17.203183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:17.203232 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:17.203551 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:17.203552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:17.203552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:17.203664 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:17.203715 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:17.204096 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 03:28:17.204180 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 03:28:17.204516 4330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:17.194049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38523 (local address 127.0.98.2:59875)
0927 03:28:17.194234 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:17.194272 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:17.195571 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:17.195830 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:17.195834 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:17.196171 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:17.196375 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:17.196380 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:17.197075 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:17.197077 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:17.197926 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:17.197928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:17.198456 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:17.198457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:17.198575 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:17.199153 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:17.199162 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:17.201103 (+ 1941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:17.202770 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:17.202779 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:17.202782 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:17.203093 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:17.203302 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:17.203303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:17.203303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:17.203483 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:17.203736 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:17.203743 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:17.203950 (+ 207us) client_negotiation.cc:769] Sending connection context
0927 03:28:17.204064 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 03:28:17.204252 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":85,"threads_started":1}
I20260927 03:28:17.205134 4328 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38523
I20260927 03:28:17.205251 4328 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:17.205438 4328 heartbeater.cc:511] Master 127.0.98.62:38523 requested a full tablet report, sending...
I20260927 03:28:17.205856 4003 ts_manager.cc:194] Registered new tserver with Master: 020a5e68f2084170aa1051a2dfbefedc (127.0.98.2:35273)
I20260927 03:28:17.206261 4003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59875
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:28:17.222608 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38523
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:17.306557 4333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:17.306774 4333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:17.306797 4333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:17.308048 4333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:17.308096 4333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:17.308125 4333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:28:17.309552 4333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:17.309815 4333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:17.310071 4333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:17.315220 4341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:17.315196 4339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:17.315420 4338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:17.315416 4333 server_base.cc:1034] running on GCE node
I20260927 03:28:17.315678 4333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:17.315888 4333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:17.317023 4333 hybrid_clock.cc:648] HybridClock initialized: now 1790479697317005 us; error 41 us; skew 500 ppm
Sep 27 03:28:17 dist-test-slave-j0hf krb5kdc[3958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:17.319666 4333 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:28:17.320127 4333 webserver.cc:460] Webserver started at http://127.0.98.3:34695/ using document root <none> and password file <none>
I20260927 03:28:17.320331 4333 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:17.320372 4333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:17.320544 4333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:17.321682 4333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-2/data/instance:
uuid: "cf2e322362484f6492e899532f04cb37"
format_stamp: "Formatted at 2026-09-27 03:28:17 on dist-test-slave-j0hf"
server_key: "97f8f2394c2440dbb1ee1ce7e4b064d1"
server_key_iv: "4c1c0e86df34ad547f4e0e922f513cb4"
server_key_version: "encryptionkey@0"
I20260927 03:28:17.321993 4333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cf2e322362484f6492e899532f04cb37"
format_stamp: "Formatted at 2026-09-27 03:28:17 on dist-test-slave-j0hf"
server_key: "97f8f2394c2440dbb1ee1ce7e4b064d1"
server_key_iv: "4c1c0e86df34ad547f4e0e922f513cb4"
server_key_version: "encryptionkey@0"
I20260927 03:28:17.323266 4333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:28:17.324072 4348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:17.324282 4333 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:28:17.324352 4333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "cf2e322362484f6492e899532f04cb37"
format_stamp: "Formatted at 2026-09-27 03:28:17 on dist-test-slave-j0hf"
server_key: "97f8f2394c2440dbb1ee1ce7e4b064d1"
server_key_iv: "4c1c0e86df34ad547f4e0e922f513cb4"
server_key_version: "encryptionkey@0"
I20260927 03:28:17.324404 4333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:17.344357 4333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:17.346817 4333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:17.346936 4333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:17.347152 4333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:17.347494 4333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:17.347550 4333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:17.347590 4333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:17.347605 4333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:17.353470 4333 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35897
I20260927 03:28:17.353524 4461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35897 every 8 connection(s)
I20260927 03:28:17.353863 4333 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:28:17.357801 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 4333
I20260927 03:28:17.357941 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790479675865103-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:28:17.358121 392 external_mini_cluster.cc:1428] Setting key bdd2d813660e6af19bc436cdce9a4efb
Sep 27 03:28:17 dist-test-slave-j0hf krb5kdc[3958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:17.364470 4061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:17.354612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56135 (local address 127.0.98.62:38523)
0927 03:28:17.354689 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:17.354692 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:17.354938 (+ 246us) server_negotiation.cc:407] Connection header received
0927 03:28:17.355984 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:17.355985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:17.355998 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:17.356065 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:17.356772 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:17.357280 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:17.358585 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:17.358894 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:17.358961 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:17.361375 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:17.361377 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:17.361377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:17.361384 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:17.363060 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:17.363468 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:17.363469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:17.363469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:17.363536 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:17.363768 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:17.363769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:17.363769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:17.363882 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:17.363982 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:17.364327 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 03:28:17.364400 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 03:28:17.364622 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:17.354581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38523 (local address 127.0.98.3:56135)
0927 03:28:17.354774 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:17.354819 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:17.355891 (+ 1072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:17.356152 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:17.356156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:17.356442 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:17.356654 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:17.356658 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:17.357374 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:17.357375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:17.358499 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:17.358503 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:17.358988 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:17.358989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:17.359143 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:17.359710 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:17.359719 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:17.361265 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:17.363139 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:17.363143 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:17.363145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:17.363357 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:17.363597 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:17.363598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:17.363598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:17.363707 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:17.363983 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:17.363986 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:17.364124 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 03:28:17.364246 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 03:28:17.364398 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":115,"threads_started":1}
I20260927 03:28:17.365231 4462 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38523
I20260927 03:28:17.365322 4462 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:17.365543 4462 heartbeater.cc:511] Master 127.0.98.62:38523 requested a full tablet report, sending...
I20260927 03:28:17.366127 4003 ts_manager.cc:194] Registered new tserver with Master: cf2e322362484f6492e899532f04cb37 (127.0.98.3:35897)
I20260927 03:28:17.366483 4003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56135
I20260927 03:28:17.369953 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 03:28:17 dist-test-slave-j0hf krb5kdc[3958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 03:28:17 dist-test-slave-j0hf krb5kdc[3958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 03:28:17.393750 4469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:17.385313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55452 (local address 127.0.98.1:44397)
0927 03:28:17.385485 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:17.385491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:17.385516 (+ 25us) server_negotiation.cc:407] Connection header received
0927 03:28:17.385543 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:17.385546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:17.385560 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:17.385635 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:17.386221 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:17.386926 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:17.387987 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:17.388595 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:17.388657 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:17.390846 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:17.390854 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:17.390854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:17.390861 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:17.392630 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:17.392955 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:17.392958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:17.392958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:17.393013 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:17.393195 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:17.393196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:17.393196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:17.393330 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:17.393445 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:17.393585 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 03:28:17.393666 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":102,"threads_started":1}
Sep 27 03:28:17 dist-test-slave-j0hf krb5kdc[3958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:17.401806 4061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:17.395104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55432 (local address 127.0.98.62:38523)
0927 03:28:17.395171 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:17.395173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:17.395182 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:28:17.395262 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:17.395263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:17.395274 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:17.395331 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:17.395884 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:17.396380 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:17.397079 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:17.397306 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:17.397382 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:17.399614 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:17.399616 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:17.399616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:17.399623 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:17.400778 (+ 1155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:17.401086 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:17.401087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:17.401087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:17.401137 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:17.401326 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:17.401327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:17.401327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:17.401486 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:17.401583 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:17.401676 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 03:28:17.401740 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 03:28:17.402084 4003 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55432
I20260927 03:28:17.402447 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 4065
I20260927 03:28:17.408052 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 4199
I20260927 03:28:17.412473 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 4333
I20260927 03:28:17.416893 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 3971
2026-09-27T03:28:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1851 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:28:17 dist-test-slave-j0hf krb5kdc[4474](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:28:17 dist-test-slave-j0hf krb5kdc[4474](info): set up 2 sockets
Sep 27 03:28:17 dist-test-slave-j0hf krb5kdc[4474](info): commencing operation
krb5kdc: starting...
W20260927 03:28:18.463478 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:28:18 dist-test-slave-j0hf krb5kdc[4474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:28:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:28:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:28:18.580482 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43851
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:18.660854 4487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:18.661121 4487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:18.661163 4487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:18.662333 4487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:28:18.662405 4487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:18.662431 4487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:18.662460 4487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:28:18.662482 4487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:28:18.664057 4487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:18.664346 4487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:18.664577 4487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:18.669368 4492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:18.669376 4495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:18.669374 4493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:18.669641 4487 server_base.cc:1034] running on GCE node
I20260927 03:28:18.669884 4487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:18.670210 4487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:18.671377 4487 hybrid_clock.cc:648] HybridClock initialized: now 1790479698671362 us; error 49 us; skew 500 ppm
Sep 27 03:28:18 dist-test-slave-j0hf krb5kdc[4474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:18.673792 4487 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:28:18.674186 4487 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:18.674260 4487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:18.674409 4487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:18.675541 4487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "a654486a606e4c72b123f253c99ae201"
format_stamp: "Formatted at 2026-09-27 03:28:18 on dist-test-slave-j0hf"
server_key: "72d2af2edf97e591651c0cd07925a854"
server_key_iv: "2399f7b8287ff89fd78318f2920ebde5"
server_key_version: "encryptionkey@0"
I20260927 03:28:18.675894 4487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "a654486a606e4c72b123f253c99ae201"
format_stamp: "Formatted at 2026-09-27 03:28:18 on dist-test-slave-j0hf"
server_key: "72d2af2edf97e591651c0cd07925a854"
server_key_iv: "2399f7b8287ff89fd78318f2920ebde5"
server_key_version: "encryptionkey@0"
I20260927 03:28:18.677163 4487 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:18.677866 4501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:18.678043 4487 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:18.678155 4487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "a654486a606e4c72b123f253c99ae201"
format_stamp: "Formatted at 2026-09-27 03:28:18 on dist-test-slave-j0hf"
server_key: "72d2af2edf97e591651c0cd07925a854"
server_key_iv: "2399f7b8287ff89fd78318f2920ebde5"
server_key_version: "encryptionkey@0"
I20260927 03:28:18.678236 4487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:18.704711 4487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:18.707204 4487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:18.707379 4487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:18.711452 4487 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43851
I20260927 03:28:18.711503 4553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43851 every 8 connection(s)
I20260927 03:28:18.711833 4487 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:28:18.712745 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:18.714993 4554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a654486a606e4c72b123f253c99ae201: Bootstrap starting.
I20260927 03:28:18.715272 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 4487
I20260927 03:28:18.715389 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:28:18.715660 392 external_mini_cluster.cc:1428] Setting key 58f88504f5bdcfbb4f3626fa530f827e
I20260927 03:28:18.715680 4554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a654486a606e4c72b123f253c99ae201: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:18.715965 4554 log.cc:824] T 00000000000000000000000000000000 P a654486a606e4c72b123f253c99ae201: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:18.716749 4554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a654486a606e4c72b123f253c99ae201: No bootstrap required, opened a new log
I20260927 03:28:18.719259 4554 raft_consensus.cc:348] T 00000000000000000000000000000000 P a654486a606e4c72b123f253c99ae201 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a654486a606e4c72b123f253c99ae201" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43851 } }
I20260927 03:28:18.719445 4554 raft_consensus.cc:374] T 00000000000000000000000000000000 P a654486a606e4c72b123f253c99ae201 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:18.719482 4554 raft_consensus.cc:729] T 00000000000000000000000000000000 P a654486a606e4c72b123f253c99ae201 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a654486a606e4c72b123f253c99ae201, State: Initialized, Role: FOLLOWER
I20260927 03:28:18.719623 4554 consensus_queue.cc:260] T 00000000000000000000000000000000 P a654486a606e4c72b123f253c99ae201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a654486a606e4c72b123f253c99ae201" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43851 } }
I20260927 03:28:18.719712 4554 raft_consensus.cc:388] T 00000000000000000000000000000000 P a654486a606e4c72b123f253c99ae201 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:28:18.719751 4554 raft_consensus.cc:482] T 00000000000000000000000000000000 P a654486a606e4c72b123f253c99ae201 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:28:18.719800 4554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a654486a606e4c72b123f253c99ae201 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:18.720475 4554 raft_consensus.cc:504] T 00000000000000000000000000000000 P a654486a606e4c72b123f253c99ae201 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a654486a606e4c72b123f253c99ae201" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43851 } }
I20260927 03:28:18.720585 4554 leader_election.cc:302] T 00000000000000000000000000000000 P a654486a606e4c72b123f253c99ae201 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a654486a606e4c72b123f253c99ae201; no voters:
I20260927 03:28:18.720754 4554 leader_election.cc:288] T 00000000000000000000000000000000 P a654486a606e4c72b123f253c99ae201 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:28:18.720824 4559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a654486a606e4c72b123f253c99ae201 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:18.720985 4554 sys_catalog.cc:564] T 00000000000000000000000000000000 P a654486a606e4c72b123f253c99ae201 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:28:18.721007 4559 raft_consensus.cc:686] T 00000000000000000000000000000000 P a654486a606e4c72b123f253c99ae201 [term 1 LEADER]: Becoming Leader. State: Replica: a654486a606e4c72b123f253c99ae201, State: Running, Role: LEADER
I20260927 03:28:18.721207 4559 consensus_queue.cc:237] T 00000000000000000000000000000000 P a654486a606e4c72b123f253c99ae201 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a654486a606e4c72b123f253c99ae201" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43851 } }
I20260927 03:28:18.721544 4560 sys_catalog.cc:455] T 00000000000000000000000000000000 P a654486a606e4c72b123f253c99ae201 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a654486a606e4c72b123f253c99ae201" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a654486a606e4c72b123f253c99ae201" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43851 } } }
I20260927 03:28:18.721640 4560 sys_catalog.cc:458] T 00000000000000000000000000000000 P a654486a606e4c72b123f253c99ae201 [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:18.722358 4574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:28:18.722400 4561 sys_catalog.cc:455] T 00000000000000000000000000000000 P a654486a606e4c72b123f253c99ae201 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a654486a606e4c72b123f253c99ae201. Latest consensus state: current_term: 1 leader_uuid: "a654486a606e4c72b123f253c99ae201" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a654486a606e4c72b123f253c99ae201" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43851 } } }
I20260927 03:28:18.722496 4561 sys_catalog.cc:458] T 00000000000000000000000000000000 P a654486a606e4c72b123f253c99ae201 [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:18.722913 4574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 03:28:18 dist-test-slave-j0hf krb5kdc[4474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:18.724475 4574 catalog_manager.cc:1347] Generated new cluster ID: 3d022f5de68441cf9336503e36a9d529
I20260927 03:28:18.724546 4574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:28:18.728065 4558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:18.716138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34474 (local address 127.0.98.62:43851)
0927 03:28:18.716263 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:18.716273 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:18.716312 (+ 39us) server_negotiation.cc:407] Connection header received
0927 03:28:18.716918 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:18.716927 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:18.716987 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:18.717135 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:18.718055 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:18.719485 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:18.720631 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:18.721266 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:18.721371 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:18.724435 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:18.724439 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:18.724440 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:18.724452 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:18.726438 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:18.726822 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:18.726825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:18.726826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:18.726932 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:18.727133 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:18.727134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:18.727134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:18.727458 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:18.727628 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:18.727707 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 03:28:18.727847 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":57,"threads_started":1}
I20260927 03:28:18.743988 4574 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:28:18.744565 4574 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:28:18.748932 4574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a654486a606e4c72b123f253c99ae201: Generated new TSK 0
I20260927 03:28:18.749140 4574 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:28:18.778789 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43851
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:18.862221 4581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:18.862448 4581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:18.862475 4581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:18.863821 4581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:18.863878 4581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:18.863906 4581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:28:18.865520 4581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:18.865808 4581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:18.866077 4581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:18.871227 4586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:18.871227 4587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:18.871227 4589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:18.871557 4581 server_base.cc:1034] running on GCE node
I20260927 03:28:18.871834 4581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:18.872092 4581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:18.873238 4581 hybrid_clock.cc:648] HybridClock initialized: now 1790479698873228 us; error 29 us; skew 500 ppm
Sep 27 03:28:18 dist-test-slave-j0hf krb5kdc[4474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:18.875960 4581 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:28:18.876332 4581 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:18.876381 4581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:18.876518 4581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:18.877629 4581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-0/data/instance:
uuid: "d6a9d12d8b7f49c69cf9a3e7b2452a1c"
format_stamp: "Formatted at 2026-09-27 03:28:18 on dist-test-slave-j0hf"
server_key: "f4a9b4717693891d10a73909b9d1e6a7"
server_key_iv: "fad48cac7f5763901cc1400730746428"
server_key_version: "encryptionkey@0"
I20260927 03:28:18.878000 4581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d6a9d12d8b7f49c69cf9a3e7b2452a1c"
format_stamp: "Formatted at 2026-09-27 03:28:18 on dist-test-slave-j0hf"
server_key: "f4a9b4717693891d10a73909b9d1e6a7"
server_key_iv: "fad48cac7f5763901cc1400730746428"
server_key_version: "encryptionkey@0"
I20260927 03:28:18.879335 4581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 03:28:18.880141 4595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:18.880337 4581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:28:18.880429 4581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "d6a9d12d8b7f49c69cf9a3e7b2452a1c"
format_stamp: "Formatted at 2026-09-27 03:28:18 on dist-test-slave-j0hf"
server_key: "f4a9b4717693891d10a73909b9d1e6a7"
server_key_iv: "fad48cac7f5763901cc1400730746428"
server_key_version: "encryptionkey@0"
I20260927 03:28:18.880506 4581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:18.900480 4581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:18.902525 4581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:18.902709 4581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:18.902968 4581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:18.903308 4581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:18.903344 4581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:18.903389 4581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:18.903443 4581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:18.909571 4581 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43607
I20260927 03:28:18.909618 4708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43607 every 8 connection(s)
I20260927 03:28:18.909960 4581 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:28:18.914100 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 4581
I20260927 03:28:18.914224 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:28:18.914422 392 external_mini_cluster.cc:1428] Setting key de839e5b5cb9a3373a8d132393fbcc8d
Sep 27 03:28:18 dist-test-slave-j0hf krb5kdc[4474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:18.921670 4558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:18.910866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35815 (local address 127.0.98.62:43851)
0927 03:28:18.911186 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:18.911188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:18.911199 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:28:18.911947 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:18.911948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:18.911961 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:18.912034 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:18.913513 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:18.914171 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:18.915361 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:18.915714 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:18.915802 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:18.918570 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:18.918572 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:18.918572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:18.918581 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:18.920083 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:18.920511 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:18.920513 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:18.920513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:18.920581 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:18.920847 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:18.920848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:18.920848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:18.921058 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:18.921119 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:18.921466 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 03:28:18.921562 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295}
I20260927 03:28:18.921871 4711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:18.910838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43851 (local address 127.0.98.1:35815)
0927 03:28:18.910996 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:18.911057 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:18.911810 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:18.912118 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:18.912122 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:18.912602 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:18.913412 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:18.913420 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:18.914292 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:18.914293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:18.915262 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:18.915265 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:18.915874 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:18.915876 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:18.916030 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:18.916814 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:18.916825 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:18.918461 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:18.920185 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:18.920192 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:18.920195 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:18.920430 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:18.920652 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:18.920653 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:18.920653 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:18.920777 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:18.921128 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:18.921135 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:18.921321 (+ 186us) client_negotiation.cc:769] Sending connection context
0927 03:28:18.921430 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 03:28:18.921603 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260927 03:28:18.922475 4709 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43851
I20260927 03:28:18.922598 4709 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:18.922773 4709 heartbeater.cc:511] Master 127.0.98.62:43851 requested a full tablet report, sending...
I20260927 03:28:18.923395 4518 ts_manager.cc:194] Registered new tserver with Master: d6a9d12d8b7f49c69cf9a3e7b2452a1c (127.0.98.1:43607)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 03:28:18.924458 4518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:28:18.942552 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43851
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:19.023867 4714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:19.024103 4714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:19.024142 4714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:19.025280 4714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:19.025350 4714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:19.025395 4714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:28:19.026903 4714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:19.027184 4714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:19.027413 4714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:19.032342 4720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:19.032349 4722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:19.032356 4719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:19.032578 4714 server_base.cc:1034] running on GCE node
I20260927 03:28:19.032874 4714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:19.033140 4714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:19.034310 4714 hybrid_clock.cc:648] HybridClock initialized: now 1790479699034290 us; error 32 us; skew 500 ppm
Sep 27 03:28:19 dist-test-slave-j0hf krb5kdc[4474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:19.036870 4714 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:28:19.037226 4714 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:19.037297 4714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:19.037449 4714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:19.038580 4714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-1/data/instance:
uuid: "022ad8ea4ec844499800da47fab29143"
format_stamp: "Formatted at 2026-09-27 03:28:19 on dist-test-slave-j0hf"
server_key: "ff322ec650faf9f95bf4a835527d8db4"
server_key_iv: "bb9d4cd55eccb4b0ef527903ce634a79"
server_key_version: "encryptionkey@0"
I20260927 03:28:19.038951 4714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-1/wal/instance:
uuid: "022ad8ea4ec844499800da47fab29143"
format_stamp: "Formatted at 2026-09-27 03:28:19 on dist-test-slave-j0hf"
server_key: "ff322ec650faf9f95bf4a835527d8db4"
server_key_iv: "bb9d4cd55eccb4b0ef527903ce634a79"
server_key_version: "encryptionkey@0"
I20260927 03:28:19.040344 4714 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:28:19.041069 4728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:19.041244 4714 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:19.041347 4714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "022ad8ea4ec844499800da47fab29143"
format_stamp: "Formatted at 2026-09-27 03:28:19 on dist-test-slave-j0hf"
server_key: "ff322ec650faf9f95bf4a835527d8db4"
server_key_iv: "bb9d4cd55eccb4b0ef527903ce634a79"
server_key_version: "encryptionkey@0"
I20260927 03:28:19.041425 4714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:19.060179 4714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:19.062282 4714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:19.062454 4714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:19.062702 4714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:19.063030 4714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:19.063086 4714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:19.063138 4714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:19.063169 4714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:19.069128 4714 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40969
I20260927 03:28:19.069175 4841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40969 every 8 connection(s)
I20260927 03:28:19.069491 4714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:28:19 dist-test-slave-j0hf krb5kdc[4474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:19.077728 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 4714
I20260927 03:28:19.077812 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:28:19.078002 392 external_mini_cluster.cc:1428] Setting key d51804ec7ad0d3d371de821f7857a79e
I20260927 03:28:19.079066 4558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:19.070233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47145 (local address 127.0.98.62:43851)
0927 03:28:19.070316 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:19.070319 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:19.070518 (+ 199us) server_negotiation.cc:407] Connection header received
0927 03:28:19.071070 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:19.071072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:19.071084 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:19.071150 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:19.072194 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:19.072697 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:19.073724 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:19.073920 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:19.074003 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:19.076183 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:19.076185 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:19.076185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:19.076193 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:19.077484 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:19.077948 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:19.077949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:19.077950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:19.078015 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:19.078358 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:19.078358 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:19.078359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:19.078468 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:19.078526 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:19.078839 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 03:28:19.078929 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 03:28:19.079077 4844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:19.070230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43851 (local address 127.0.98.2:47145)
0927 03:28:19.070383 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:19.070428 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:19.070950 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:19.071224 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:19.071228 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:19.071585 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:19.072085 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:19.072091 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:19.072795 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:19.072796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:19.073599 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:19.073601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:19.074013 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:19.074014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:19.074141 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:19.074620 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:19.074628 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:19.076068 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:19.077592 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:19.077597 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:19.077599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:19.077857 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:19.078085 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:19.078086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:19.078086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:19.078242 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:19.078541 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:19.078547 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:19.078698 (+ 151us) client_negotiation.cc:769] Sending connection context
0927 03:28:19.078791 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 03:28:19.078912 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
I20260927 03:28:19.079710 4842 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43851
I20260927 03:28:19.079820 4842 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:19.079973 4842 heartbeater.cc:511] Master 127.0.98.62:43851 requested a full tablet report, sending...
I20260927 03:28:19.080425 4518 ts_manager.cc:194] Registered new tserver with Master: 022ad8ea4ec844499800da47fab29143 (127.0.98.2:40969)
I20260927 03:28:19.080811 4518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47145
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:28:19.102540 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43851
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:19.186321 4847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:19.186540 4847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:19.186564 4847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:19.187908 4847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:19.187952 4847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:19.188001 4847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:28:19.189590 4847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:19.189851 4847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:19.190094 4847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:19.195348 4852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:19.195319 4855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:19.195313 4853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:19.195700 4847 server_base.cc:1034] running on GCE node
I20260927 03:28:19.196007 4847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:19.196265 4847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:19.197434 4847 hybrid_clock.cc:648] HybridClock initialized: now 1790479699197400 us; error 47 us; skew 500 ppm
Sep 27 03:28:19 dist-test-slave-j0hf krb5kdc[4474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:19.199932 4847 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:28:19.200372 4847 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:19.200452 4847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:19.200591 4847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:19.201686 4847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-2/data/instance:
uuid: "5f46a69096f64964ad64c3a015c955e0"
format_stamp: "Formatted at 2026-09-27 03:28:19 on dist-test-slave-j0hf"
server_key: "bef880497143849a77b92350674a4d97"
server_key_iv: "a3331e28ea24bf83d5e40d6401eb585c"
server_key_version: "encryptionkey@0"
I20260927 03:28:19.202071 4847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5f46a69096f64964ad64c3a015c955e0"
format_stamp: "Formatted at 2026-09-27 03:28:19 on dist-test-slave-j0hf"
server_key: "bef880497143849a77b92350674a4d97"
server_key_iv: "a3331e28ea24bf83d5e40d6401eb585c"
server_key_version: "encryptionkey@0"
I20260927 03:28:19.203342 4847 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:28:19.204123 4861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:19.204293 4847 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:19.204399 4847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "5f46a69096f64964ad64c3a015c955e0"
format_stamp: "Formatted at 2026-09-27 03:28:19 on dist-test-slave-j0hf"
server_key: "bef880497143849a77b92350674a4d97"
server_key_iv: "a3331e28ea24bf83d5e40d6401eb585c"
server_key_version: "encryptionkey@0"
I20260927 03:28:19.204483 4847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:19.216898 4847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:19.219553 4847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:19.219733 4847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:19.219981 4847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:19.220309 4847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:19.220363 4847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:19.220407 4847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:19.220433 4847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:19.226687 4847 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34703
I20260927 03:28:19.226723 4974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34703 every 8 connection(s)
I20260927 03:28:19.227018 4847 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:28:19.227321 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 4847
I20260927 03:28:19.227454 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:28:19.227653 392 external_mini_cluster.cc:1428] Setting key 94d2aa635b69aeb05d93097a4d6067bd
Sep 27 03:28:19 dist-test-slave-j0hf krb5kdc[4474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:19.236328 4558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:19.227779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40731 (local address 127.0.98.62:43851)
0927 03:28:19.227834 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:19.227837 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:19.228100 (+ 263us) server_negotiation.cc:407] Connection header received
0927 03:28:19.228809 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:19.228811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:19.228824 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:19.228894 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:19.229801 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:19.230293 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:19.231263 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:19.231522 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:19.231576 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:19.233688 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:19.233690 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:19.233690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:19.233697 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:19.235123 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:19.235517 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:19.235518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:19.235519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:19.235581 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:19.235810 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:19.235811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:19.235811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:19.235916 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:19.235966 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:19.236201 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 03:28:19.236260 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:28:19.236557 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:19.227771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43851 (local address 127.0.98.3:40731)
0927 03:28:19.227963 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:19.228013 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:19.228699 (+ 686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:19.228951 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:19.228954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:19.229222 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:19.229719 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:19.229725 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:19.230375 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:19.230375 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:19.231188 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:19.231189 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:19.231588 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:19.231588 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:19.231694 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:19.232165 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:19.232173 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:19.233579 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:19.235192 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:19.235197 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:19.235198 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:19.235399 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:19.235653 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:19.235654 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:19.235654 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:19.235755 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:19.235970 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:19.235973 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:19.236090 (+ 117us) client_negotiation.cc:769] Sending connection context
0927 03:28:19.236175 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 03:28:19.236296 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":93,"threads_started":1}
I20260927 03:28:19.237125 4975 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43851
I20260927 03:28:19.237237 4975 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:19.237393 4975 heartbeater.cc:511] Master 127.0.98.62:43851 requested a full tablet report, sending...
I20260927 03:28:19.237823 4518 ts_manager.cc:194] Registered new tserver with Master: 5f46a69096f64964ad64c3a015c955e0 (127.0.98.3:34703)
I20260927 03:28:19.238194 4518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40731
I20260927 03:28:19.239408 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:28:19.246928 4558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:19.240697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34486 (local address 127.0.98.62:43851)
0927 03:28:19.240778 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:19.240780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:19.240868 (+ 88us) server_negotiation.cc:407] Connection header received
0927 03:28:19.240931 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:19.240932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:19.240943 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:19.240998 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:19.241560 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:19.242049 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:19.242834 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:19.243036 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:19.243145 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:19.244470 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:19.244471 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:19.244472 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:19.244479 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:19.245730 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:19.246094 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:19.246095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:19.246096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:19.246152 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:19.246356 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:19.246357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:19.246357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:19.246520 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:19.246633 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:19.246742 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 03:28:19.246825 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 03:28:19.248551 4518 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:28:19.250478 4518 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:28:19.257688 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:19.252869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34703 (local address 127.0.0.1:42468)
0927 03:28:19.253246 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:19.253256 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:19.253307 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:19.253514 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:19.253515 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:19.253516 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:19.253594 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:19.253596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:19.254984 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:19.254985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:19.256433 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:19.256436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:19.257353 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:19.257354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:19.257444 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:19.257460 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 03:28:19.257539 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 03:28:19.257603 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":32,"thread_start_us":86,"threads_started":1}
I20260927 03:28:19.257694 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:19.253010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42468 (local address 127.0.98.3:34703)
0927 03:28:19.253177 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:19.253180 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:19.253322 (+ 142us) server_negotiation.cc:407] Connection header received
0927 03:28:19.253370 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:19.253373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:19.253391 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:19.253695 (+ 304us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:19.253781 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:19.254875 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:19.256522 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:19.257271 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:19.257351 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:19.257419 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:19.257567 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 03:28:19.257611 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":77,"threads_started":1}
I20260927 03:28:19.258524 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:19.253045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40969 (local address 127.0.0.1:49338)
0927 03:28:19.253380 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:19.253389 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:19.253422 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:19.255209 (+ 1787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:19.255210 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:19.255213 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:19.255262 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:19.255264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:19.256277 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:19.256279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:19.257334 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:19.257336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:19.258319 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:19.258320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:19.258375 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:19.258379 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:28:19.258421 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 03:28:19.258457 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":66,"threads_started":1}
I20260927 03:28:19.258713 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:19.252943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35600 (local address 127.0.98.1:43607)
0927 03:28:19.253114 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:19.253119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:19.253342 (+ 223us) server_negotiation.cc:407] Connection header received
0927 03:28:19.253389 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:19.253392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:19.253410 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:19.253480 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:19.253697 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:19.254875 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:19.257454 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:19.258313 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:19.258377 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:19.258435 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:19.258602 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 03:28:19.258638 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":76,"threads_started":1}
I20260927 03:28:19.258699 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:19.252846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43607 (local address 127.0.0.1:35600)
0927 03:28:19.253131 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:19.253174 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:19.253248 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:19.253502 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:19.253506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:19.253509 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:19.253594 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:19.253597 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:19.254985 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:19.254986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:19.256433 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:19.256436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:19.258462 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:19.258463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:19.258516 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:19.258519 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:28:19.258588 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 03:28:19.258630 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":63,"thread_start_us":58,"threads_started":1}
I20260927 03:28:19.258817 4909 tablet_service.cc:1467] Processing CreateTablet for tablet e89aa8d78506400299ddd7b23bf53518 (DEFAULT_TABLE table=test-table [id=986ffdb2faef45cbb7e301efe05b81f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:19.259199 4909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e89aa8d78506400299ddd7b23bf53518. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:19.259539 4643 tablet_service.cc:1467] Processing CreateTablet for tablet e89aa8d78506400299ddd7b23bf53518 (DEFAULT_TABLE table=test-table [id=986ffdb2faef45cbb7e301efe05b81f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:19.259828 4643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e89aa8d78506400299ddd7b23bf53518. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:19.262068 4992 tablet_bootstrap.cc:492] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0: Bootstrap starting.
I20260927 03:28:19.262341 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:19.253266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49338 (local address 127.0.98.2:40969)
0927 03:28:19.255054 (+ 1788us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:19.255058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:19.255065 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:28:19.255097 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:19.255098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:19.255111 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:19.255197 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:19.255318 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:19.256181 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:19.257416 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:19.258218 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:19.258317 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:19.262178 (+ 3861us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:19.262210 (+ 32us) server_negotiation.cc:299] Negotiation successful
0927 03:28:19.262264 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1745,"thread_start_us":124,"threads_started":1}
I20260927 03:28:19.262740 4993 tablet_bootstrap.cc:492] T e89aa8d78506400299ddd7b23bf53518 P d6a9d12d8b7f49c69cf9a3e7b2452a1c: Bootstrap starting.
I20260927 03:28:19.262807 4992 tablet_bootstrap.cc:654] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:19.263013 4776 tablet_service.cc:1467] Processing CreateTablet for tablet e89aa8d78506400299ddd7b23bf53518 (DEFAULT_TABLE table=test-table [id=986ffdb2faef45cbb7e301efe05b81f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:19.263315 4776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e89aa8d78506400299ddd7b23bf53518. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:19.263455 4993 tablet_bootstrap.cc:654] T e89aa8d78506400299ddd7b23bf53518 P d6a9d12d8b7f49c69cf9a3e7b2452a1c: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:19.263448 4992 log.cc:824] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:19.264020 4993 log.cc:824] T e89aa8d78506400299ddd7b23bf53518 P d6a9d12d8b7f49c69cf9a3e7b2452a1c: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:19.264417 4992 tablet_bootstrap.cc:492] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0: No bootstrap required, opened a new log
I20260927 03:28:19.264557 4992 ts_tablet_manager.cc:1397] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 03:28:19.264743 4993 tablet_bootstrap.cc:492] T e89aa8d78506400299ddd7b23bf53518 P d6a9d12d8b7f49c69cf9a3e7b2452a1c: No bootstrap required, opened a new log
I20260927 03:28:19.264807 4993 ts_tablet_manager.cc:1397] T e89aa8d78506400299ddd7b23bf53518 P d6a9d12d8b7f49c69cf9a3e7b2452a1c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 03:28:19.266366 4996 tablet_bootstrap.cc:492] T e89aa8d78506400299ddd7b23bf53518 P 022ad8ea4ec844499800da47fab29143: Bootstrap starting.
I20260927 03:28:19.266588 4992 raft_consensus.cc:348] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a9d12d8b7f49c69cf9a3e7b2452a1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43607 } } peers { permanent_uuid: "022ad8ea4ec844499800da47fab29143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40969 } } peers { permanent_uuid: "5f46a69096f64964ad64c3a015c955e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34703 } }
I20260927 03:28:19.266772 4992 raft_consensus.cc:374] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:19.266805 4992 raft_consensus.cc:729] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f46a69096f64964ad64c3a015c955e0, State: Initialized, Role: FOLLOWER
I20260927 03:28:19.266734 4993 raft_consensus.cc:348] T e89aa8d78506400299ddd7b23bf53518 P d6a9d12d8b7f49c69cf9a3e7b2452a1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a9d12d8b7f49c69cf9a3e7b2452a1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43607 } } peers { permanent_uuid: "022ad8ea4ec844499800da47fab29143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40969 } } peers { permanent_uuid: "5f46a69096f64964ad64c3a015c955e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34703 } }
I20260927 03:28:19.266856 4996 tablet_bootstrap.cc:654] T e89aa8d78506400299ddd7b23bf53518 P 022ad8ea4ec844499800da47fab29143: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:19.266870 4993 raft_consensus.cc:374] T e89aa8d78506400299ddd7b23bf53518 P d6a9d12d8b7f49c69cf9a3e7b2452a1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:19.266903 4993 raft_consensus.cc:729] T e89aa8d78506400299ddd7b23bf53518 P d6a9d12d8b7f49c69cf9a3e7b2452a1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a9d12d8b7f49c69cf9a3e7b2452a1c, State: Initialized, Role: FOLLOWER
I20260927 03:28:19.266928 4992 consensus_queue.cc:260] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a9d12d8b7f49c69cf9a3e7b2452a1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43607 } } peers { permanent_uuid: "022ad8ea4ec844499800da47fab29143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40969 } } peers { permanent_uuid: "5f46a69096f64964ad64c3a015c955e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34703 } }
I20260927 03:28:19.266991 4993 consensus_queue.cc:260] T e89aa8d78506400299ddd7b23bf53518 P d6a9d12d8b7f49c69cf9a3e7b2452a1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a9d12d8b7f49c69cf9a3e7b2452a1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43607 } } peers { permanent_uuid: "022ad8ea4ec844499800da47fab29143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40969 } } peers { permanent_uuid: "5f46a69096f64964ad64c3a015c955e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34703 } }
I20260927 03:28:19.267216 4992 ts_tablet_manager.cc:1428] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:28:19.267237 4993 ts_tablet_manager.cc:1428] T e89aa8d78506400299ddd7b23bf53518 P d6a9d12d8b7f49c69cf9a3e7b2452a1c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 03:28:19.267244 4975 heartbeater.cc:503] Master 127.0.98.62:43851 was elected leader, sending a full tablet report...
I20260927 03:28:19.267333 4709 heartbeater.cc:503] Master 127.0.98.62:43851 was elected leader, sending a full tablet report...
I20260927 03:28:19.267490 4996 log.cc:824] T e89aa8d78506400299ddd7b23bf53518 P 022ad8ea4ec844499800da47fab29143: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:19.268800 4996 tablet_bootstrap.cc:492] T e89aa8d78506400299ddd7b23bf53518 P 022ad8ea4ec844499800da47fab29143: No bootstrap required, opened a new log
I20260927 03:28:19.268860 4996 ts_tablet_manager.cc:1397] T e89aa8d78506400299ddd7b23bf53518 P 022ad8ea4ec844499800da47fab29143: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 03:28:19.270275 4996 raft_consensus.cc:348] T e89aa8d78506400299ddd7b23bf53518 P 022ad8ea4ec844499800da47fab29143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a9d12d8b7f49c69cf9a3e7b2452a1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43607 } } peers { permanent_uuid: "022ad8ea4ec844499800da47fab29143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40969 } } peers { permanent_uuid: "5f46a69096f64964ad64c3a015c955e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34703 } }
I20260927 03:28:19.270411 4996 raft_consensus.cc:374] T e89aa8d78506400299ddd7b23bf53518 P 022ad8ea4ec844499800da47fab29143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:19.270433 4996 raft_consensus.cc:729] T e89aa8d78506400299ddd7b23bf53518 P 022ad8ea4ec844499800da47fab29143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 022ad8ea4ec844499800da47fab29143, State: Initialized, Role: FOLLOWER
I20260927 03:28:19.270553 4996 consensus_queue.cc:260] T e89aa8d78506400299ddd7b23bf53518 P 022ad8ea4ec844499800da47fab29143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a9d12d8b7f49c69cf9a3e7b2452a1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43607 } } peers { permanent_uuid: "022ad8ea4ec844499800da47fab29143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40969 } } peers { permanent_uuid: "5f46a69096f64964ad64c3a015c955e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34703 } }
I20260927 03:28:19.270826 4996 ts_tablet_manager.cc:1428] T e89aa8d78506400299ddd7b23bf53518 P 022ad8ea4ec844499800da47fab29143: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:28:19.270861 4842 heartbeater.cc:503] Master 127.0.98.62:43851 was elected leader, sending a full tablet report...
W20260927 03:28:19.320034 4843 tablet.cc:2369] T e89aa8d78506400299ddd7b23bf53518 P 022ad8ea4ec844499800da47fab29143: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:28:19.368777 4997 raft_consensus.cc:482] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:28:19.368958 4997 raft_consensus.cc:504] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a9d12d8b7f49c69cf9a3e7b2452a1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43607 } } peers { permanent_uuid: "022ad8ea4ec844499800da47fab29143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40969 } } peers { permanent_uuid: "5f46a69096f64964ad64c3a015c955e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34703 } }
I20260927 03:28:19.369364 4997 leader_election.cc:288] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6a9d12d8b7f49c69cf9a3e7b2452a1c (127.0.98.1:43607), 022ad8ea4ec844499800da47fab29143 (127.0.98.2:40969)
I20260927 03:28:19.372357 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:19.369639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40969 (local address 127.0.98.3:33355)
0927 03:28:19.369847 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:19.369858 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:19.369908 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:19.370061 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:19.370061 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:19.370063 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:19.370132 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:19.370133 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:19.370748 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:19.370749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:19.371540 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:19.371542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:19.372094 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:19.372095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:19.372174 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:19.372178 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:28:19.372234 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 03:28:19.372264 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":120,"threads_started":1}
I20260927 03:28:19.372368 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:19.369859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33355 (local address 127.0.98.2:40969)
0927 03:28:19.369903 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:19.369905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:19.369914 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:28:19.369978 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:19.369978 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:19.369991 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:19.370040 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:19.370205 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:19.370636 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:19.371628 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:19.372008 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:19.372062 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:19.372069 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:19.372251 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 03:28:19.372289 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:28:19.372494 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:19.369668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36035 (local address 127.0.98.1:43607)
0927 03:28:19.369736 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:19.369738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:19.369821 (+ 83us) server_negotiation.cc:407] Connection header received
0927 03:28:19.369849 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:19.369850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:19.369866 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:19.369923 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:19.370084 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:19.370612 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:19.371693 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:19.372075 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:19.372154 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:19.372166 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:19.372366 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 03:28:19.372413 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 03:28:19.372469 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:19.369538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43607 (local address 127.0.98.3:36035)
0927 03:28:19.369713 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:19.369727 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:19.369773 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:19.369928 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:19.369929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:19.369931 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:19.370007 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:19.370009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:19.370702 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:19.370703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:19.371548 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:19.371551 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:19.372175 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:19.372176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:19.372275 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:19.372278 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:28:19.372336 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 03:28:19.372371 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":10}
I20260927 03:28:19.372927 4663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89aa8d78506400299ddd7b23bf53518" candidate_uuid: "5f46a69096f64964ad64c3a015c955e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a9d12d8b7f49c69cf9a3e7b2452a1c" is_pre_election: true
I20260927 03:28:19.372915 4796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89aa8d78506400299ddd7b23bf53518" candidate_uuid: "5f46a69096f64964ad64c3a015c955e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "022ad8ea4ec844499800da47fab29143" is_pre_election: true
I20260927 03:28:19.373078 4796 raft_consensus.cc:2454] T e89aa8d78506400299ddd7b23bf53518 P 022ad8ea4ec844499800da47fab29143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46a69096f64964ad64c3a015c955e0 in term 0.
I20260927 03:28:19.373078 4663 raft_consensus.cc:2454] T e89aa8d78506400299ddd7b23bf53518 P d6a9d12d8b7f49c69cf9a3e7b2452a1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f46a69096f64964ad64c3a015c955e0 in term 0.
I20260927 03:28:19.373292 4864 leader_election.cc:302] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f46a69096f64964ad64c3a015c955e0, d6a9d12d8b7f49c69cf9a3e7b2452a1c; no voters:
I20260927 03:28:19.373418 4997 raft_consensus.cc:2781] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:28:19.373474 4997 raft_consensus.cc:482] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:28:19.373499 4997 raft_consensus.cc:3037] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:19.374205 4997 raft_consensus.cc:504] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a9d12d8b7f49c69cf9a3e7b2452a1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43607 } } peers { permanent_uuid: "022ad8ea4ec844499800da47fab29143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40969 } } peers { permanent_uuid: "5f46a69096f64964ad64c3a015c955e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34703 } }
I20260927 03:28:19.374338 4997 leader_election.cc:288] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0 [CANDIDATE]: Term 1 election: Requested vote from peers d6a9d12d8b7f49c69cf9a3e7b2452a1c (127.0.98.1:43607), 022ad8ea4ec844499800da47fab29143 (127.0.98.2:40969)
I20260927 03:28:19.374529 4663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89aa8d78506400299ddd7b23bf53518" candidate_uuid: "5f46a69096f64964ad64c3a015c955e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a9d12d8b7f49c69cf9a3e7b2452a1c"
I20260927 03:28:19.374529 4796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89aa8d78506400299ddd7b23bf53518" candidate_uuid: "5f46a69096f64964ad64c3a015c955e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "022ad8ea4ec844499800da47fab29143"
I20260927 03:28:19.374610 4796 raft_consensus.cc:3037] T e89aa8d78506400299ddd7b23bf53518 P 022ad8ea4ec844499800da47fab29143 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:19.374610 4663 raft_consensus.cc:3037] T e89aa8d78506400299ddd7b23bf53518 P d6a9d12d8b7f49c69cf9a3e7b2452a1c [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:19.375522 4663 raft_consensus.cc:2454] T e89aa8d78506400299ddd7b23bf53518 P d6a9d12d8b7f49c69cf9a3e7b2452a1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46a69096f64964ad64c3a015c955e0 in term 1.
I20260927 03:28:19.375522 4796 raft_consensus.cc:2454] T e89aa8d78506400299ddd7b23bf53518 P 022ad8ea4ec844499800da47fab29143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f46a69096f64964ad64c3a015c955e0 in term 1.
I20260927 03:28:19.375694 4864 leader_election.cc:302] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 022ad8ea4ec844499800da47fab29143, 5f46a69096f64964ad64c3a015c955e0; no voters:
I20260927 03:28:19.375795 4997 raft_consensus.cc:2781] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:19.375946 4997 raft_consensus.cc:686] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0 [term 1 LEADER]: Becoming Leader. State: Replica: 5f46a69096f64964ad64c3a015c955e0, State: Running, Role: LEADER
I20260927 03:28:19.376047 4997 consensus_queue.cc:237] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a9d12d8b7f49c69cf9a3e7b2452a1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43607 } } peers { permanent_uuid: "022ad8ea4ec844499800da47fab29143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40969 } } peers { permanent_uuid: "5f46a69096f64964ad64c3a015c955e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34703 } }
I20260927 03:28:19.376937 4517 catalog_manager.cc:5495] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f46a69096f64964ad64c3a015c955e0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5f46a69096f64964ad64c3a015c955e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a9d12d8b7f49c69cf9a3e7b2452a1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "022ad8ea4ec844499800da47fab29143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f46a69096f64964ad64c3a015c955e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34703 } health_report { overall_health: HEALTHY } } }
I20260927 03:28:19.402383 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:19.400209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42480 (local address 127.0.98.3:34703)
0927 03:28:19.400269 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:19.400273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:19.400286 (+ 13us) server_negotiation.cc:407] Connection header received
0927 03:28:19.400319 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:19.400320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:19.400338 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:19.400399 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:28:19.400524 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:19.400975 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:19.401688 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:19.401859 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:19.401905 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:19.402035 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:28:19.402139 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:28:19.402197 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:19.402235 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 03:28:19.402251 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:28:19.404629 4663 raft_consensus.cc:1261] T e89aa8d78506400299ddd7b23bf53518 P d6a9d12d8b7f49c69cf9a3e7b2452a1c [term 1 FOLLOWER]: Refusing update from remote peer 5f46a69096f64964ad64c3a015c955e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:19.404630 4796 raft_consensus.cc:1261] T e89aa8d78506400299ddd7b23bf53518 P 022ad8ea4ec844499800da47fab29143 [term 1 FOLLOWER]: Refusing update from remote peer 5f46a69096f64964ad64c3a015c955e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:19.404999 5002 consensus_queue.cc:1035] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6a9d12d8b7f49c69cf9a3e7b2452a1c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:19.405117 4997 consensus_queue.cc:1035] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "022ad8ea4ec844499800da47fab29143" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:19.407382 5005 mvcc.cc:205] Tried to move back new op lower bound from 7333804848758218752 to 7333804848645021696. Current Snapshot: MvccSnapshot[applied={T|T < 7333804848758218752}]
I20260927 03:28:19.408107 5007 mvcc.cc:205] Tried to move back new op lower bound from 7333804848758218752 to 7333804848645021696. Current Snapshot: MvccSnapshot[applied={T|T < 7333804848758218752}]
I20260927 03:28:19.408386 5006 mvcc.cc:205] Tried to move back new op lower bound from 7333804848758218752 to 7333804848645021696. Current Snapshot: MvccSnapshot[applied={T|T < 7333804848758218752}]
I20260927 03:28:19.410274 4517 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:28:19.410365 4517 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:28:19.411089 4517 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a654486a606e4c72b123f253c99ae201: Sending DeleteTablet for 3 replicas of tablet e89aa8d78506400299ddd7b23bf53518
I20260927 03:28:19.411413 4909 tablet_service.cc:1514] Processing DeleteTablet for tablet e89aa8d78506400299ddd7b23bf53518 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42468
I20260927 03:28:19.411473 4643 tablet_service.cc:1514] Processing DeleteTablet for tablet e89aa8d78506400299ddd7b23bf53518 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35600
I20260927 03:28:19.411710 4776 tablet_service.cc:1514] Processing DeleteTablet for tablet e89aa8d78506400299ddd7b23bf53518 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49338
I20260927 03:28:19.411819 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 4581
I20260927 03:28:19.413477 5017 tablet_replica.cc:330] T e89aa8d78506400299ddd7b23bf53518 P 022ad8ea4ec844499800da47fab29143: stopping tablet replica
I20260927 03:28:19.414561 5015 tablet_replica.cc:330] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0: stopping tablet replica
I20260927 03:28:19.414681 5015 raft_consensus.cc:2229] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:28:19.414848 5015 raft_consensus.cc:2258] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:19.415637 5017 raft_consensus.cc:2229] T e89aa8d78506400299ddd7b23bf53518 P 022ad8ea4ec844499800da47fab29143 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:28:19.415822 5017 raft_consensus.cc:2258] T e89aa8d78506400299ddd7b23bf53518 P 022ad8ea4ec844499800da47fab29143 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:19.415899 5015 ts_tablet_manager.cc:1905] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:19.416337 5017 ts_tablet_manager.cc:1905] T e89aa8d78506400299ddd7b23bf53518 P 022ad8ea4ec844499800da47fab29143: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 03:28:19.417531 4504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43607 (error 108)
I20260927 03:28:19.417930 5017 ts_tablet_manager.cc:1918] T e89aa8d78506400299ddd7b23bf53518 P 022ad8ea4ec844499800da47fab29143: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:28:19.418053 5015 ts_tablet_manager.cc:1918] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:28:19.418100 5017 log.cc:1196] T e89aa8d78506400299ddd7b23bf53518 P 022ad8ea4ec844499800da47fab29143: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-1/wal/wals/e89aa8d78506400299ddd7b23bf53518
I20260927 03:28:19.418222 5015 log.cc:1196] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790479675865103-392-0/minicluster-data/ts-2/wal/wals/e89aa8d78506400299ddd7b23bf53518
I20260927 03:28:19.418424 5017 ts_tablet_manager.cc:1939] T e89aa8d78506400299ddd7b23bf53518 P 022ad8ea4ec844499800da47fab29143: Deleting consensus metadata
I20260927 03:28:19.418515 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:19.418387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43607 (local address 127.0.0.1:35610)
0927 03:28:19.418439 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:19.418474 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 03:28:19.418668 5015 ts_tablet_manager.cc:1939] T e89aa8d78506400299ddd7b23bf53518 P 5f46a69096f64964ad64c3a015c955e0: Deleting consensus metadata
W20260927 03:28:19.418645 4504 catalog_manager.cc:4584] TS d6a9d12d8b7f49c69cf9a3e7b2452a1c (127.0.98.1:43607): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e89aa8d78506400299ddd7b23bf53518: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43607: connect: Connection refused (error 111)
I20260927 03:28:19.418828 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 4714
I20260927 03:28:19.418920 4504 catalog_manager.cc:4841] TS 022ad8ea4ec844499800da47fab29143 (127.0.98.2:40969): tablet e89aa8d78506400299ddd7b23bf53518 (table test-table [id=986ffdb2faef45cbb7e301efe05b81f0]) successfully deleted
I20260927 03:28:19.419703 4502 catalog_manager.cc:4841] TS 5f46a69096f64964ad64c3a015c955e0 (127.0.98.3:34703): tablet e89aa8d78506400299ddd7b23bf53518 (table test-table [id=986ffdb2faef45cbb7e301efe05b81f0]) successfully deleted
I20260927 03:28:19.426023 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 4847
I20260927 03:28:19.430253 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 4487
2026-09-27T03:28:19Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2014 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T03:28:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:28:19Z Disabled control of system clock
I20260927 03:28:19.458424 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34173
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 03:28:19.538767 5022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:19.538992 5022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:19.539014 5022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:19.540228 5022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:28:19.540272 5022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:19.540285 5022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:19.540299 5022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:28:19.540311 5022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:28:19.541713 5022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34173
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:19.542006 5022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:19.542325 5022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:19.547372 5030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:19.547350 5027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:19.547469 5022 server_base.cc:1034] running on GCE node
W20260927 03:28:19.547354 5028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:19.547859 5022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:19.548168 5022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:19.549312 5022 hybrid_clock.cc:648] HybridClock initialized: now 1790479699549304 us; error 53 us; skew 500 ppm
I20260927 03:28:19.550611 5022 webserver.cc:460] Webserver started at http://127.0.98.62:44055/ using document root <none> and password file <none>
I20260927 03:28:19.550854 5022 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:19.550900 5022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:19.551057 5022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:19.552207 5022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "401a21beba23407e90db6ee2329a4511"
format_stamp: "Formatted at 2026-09-27 03:28:19 on dist-test-slave-j0hf"
server_key: "3b5581897a3721be20ba00bd2d4258d5"
server_key_iv: "a1cdc862a18518c5a25b73bcc0c7aa9d"
server_key_version: "encryptionkey@0"
I20260927 03:28:19.552562 5022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "401a21beba23407e90db6ee2329a4511"
format_stamp: "Formatted at 2026-09-27 03:28:19 on dist-test-slave-j0hf"
server_key: "3b5581897a3721be20ba00bd2d4258d5"
server_key_iv: "a1cdc862a18518c5a25b73bcc0c7aa9d"
server_key_version: "encryptionkey@0"
I20260927 03:28:19.553857 5022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:28:19.554579 5036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:19.554754 5022 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:19.554860 5022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "401a21beba23407e90db6ee2329a4511"
format_stamp: "Formatted at 2026-09-27 03:28:19 on dist-test-slave-j0hf"
server_key: "3b5581897a3721be20ba00bd2d4258d5"
server_key_iv: "a1cdc862a18518c5a25b73bcc0c7aa9d"
server_key_version: "encryptionkey@0"
I20260927 03:28:19.554935 5022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:19.558331 5022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:19.558600 5022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:19.558748 5022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:19.562570 5022 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34173
I20260927 03:28:19.562602 5088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34173 every 8 connection(s)
I20260927 03:28:19.562934 5022 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:28:19.563642 5089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:19.566360 5089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 401a21beba23407e90db6ee2329a4511: Bootstrap starting.
I20260927 03:28:19.566910 5089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 401a21beba23407e90db6ee2329a4511: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:19.567171 5089 log.cc:824] T 00000000000000000000000000000000 P 401a21beba23407e90db6ee2329a4511: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:19.567863 5089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 401a21beba23407e90db6ee2329a4511: No bootstrap required, opened a new log
I20260927 03:28:19.569362 5089 raft_consensus.cc:348] T 00000000000000000000000000000000 P 401a21beba23407e90db6ee2329a4511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401a21beba23407e90db6ee2329a4511" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34173 } }
I20260927 03:28:19.569500 5089 raft_consensus.cc:374] T 00000000000000000000000000000000 P 401a21beba23407e90db6ee2329a4511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:19.569523 5089 raft_consensus.cc:729] T 00000000000000000000000000000000 P 401a21beba23407e90db6ee2329a4511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401a21beba23407e90db6ee2329a4511, State: Initialized, Role: FOLLOWER
I20260927 03:28:19.569680 5089 consensus_queue.cc:260] T 00000000000000000000000000000000 P 401a21beba23407e90db6ee2329a4511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401a21beba23407e90db6ee2329a4511" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34173 } }
I20260927 03:28:19.569761 5089 raft_consensus.cc:388] T 00000000000000000000000000000000 P 401a21beba23407e90db6ee2329a4511 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:28:19.569784 5089 raft_consensus.cc:482] T 00000000000000000000000000000000 P 401a21beba23407e90db6ee2329a4511 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:28:19.569823 5089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 401a21beba23407e90db6ee2329a4511 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:19.570416 5089 raft_consensus.cc:504] T 00000000000000000000000000000000 P 401a21beba23407e90db6ee2329a4511 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401a21beba23407e90db6ee2329a4511" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34173 } }
I20260927 03:28:19.570508 5089 leader_election.cc:302] T 00000000000000000000000000000000 P 401a21beba23407e90db6ee2329a4511 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 401a21beba23407e90db6ee2329a4511; no voters:
I20260927 03:28:19.570645 5089 leader_election.cc:288] T 00000000000000000000000000000000 P 401a21beba23407e90db6ee2329a4511 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:28:19.570750 5092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 401a21beba23407e90db6ee2329a4511 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:19.570945 5092 raft_consensus.cc:686] T 00000000000000000000000000000000 P 401a21beba23407e90db6ee2329a4511 [term 1 LEADER]: Becoming Leader. State: Replica: 401a21beba23407e90db6ee2329a4511, State: Running, Role: LEADER
I20260927 03:28:19.570957 5089 sys_catalog.cc:564] T 00000000000000000000000000000000 P 401a21beba23407e90db6ee2329a4511 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:28:19.571048 5092 consensus_queue.cc:237] T 00000000000000000000000000000000 P 401a21beba23407e90db6ee2329a4511 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401a21beba23407e90db6ee2329a4511" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34173 } }
I20260927 03:28:19.572479 5093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401a21beba23407e90db6ee2329a4511 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "401a21beba23407e90db6ee2329a4511" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401a21beba23407e90db6ee2329a4511" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34173 } } }
I20260927 03:28:19.572549 5093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401a21beba23407e90db6ee2329a4511 [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:19.572448 5094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401a21beba23407e90db6ee2329a4511 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 401a21beba23407e90db6ee2329a4511. Latest consensus state: current_term: 1 leader_uuid: "401a21beba23407e90db6ee2329a4511" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401a21beba23407e90db6ee2329a4511" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34173 } } }
I20260927 03:28:19.572633 5094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401a21beba23407e90db6ee2329a4511 [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:19.572691 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 5022
I20260927 03:28:19.572790 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:28:19.572860 5102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:28:19.572996 392 external_mini_cluster.cc:1428] Setting key 117faba3501d0b940a902a97076872ff
I20260927 03:28:19.573413 5102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:28:19.575142 5102 catalog_manager.cc:1347] Generated new cluster ID: f6b0b87580df47368291b8f2c8e7f879
I20260927 03:28:19.575207 5102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:28:19.576339 5111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:19.573627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59828 (local address 127.0.98.62:34173)
0927 03:28:19.574240 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:19.574250 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:19.574495 (+ 245us) server_negotiation.cc:407] Connection header received
0927 03:28:19.574736 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:19.574745 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:19.574796 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:19.574964 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:19.575889 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:19.575893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:19.575894 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:28:19.575898 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:19.575948 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:28:19.575963 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:19.576026 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:19.576115 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 03:28:19.576172 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":68,"threads_started":1}
I20260927 03:28:19.582602 5102 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:28:19.582782 5102 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:28:19.593739 5102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 401a21beba23407e90db6ee2329a4511: Generated new TSK 0
I20260927 03:28:19.594043 5102 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:28:19.601361 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34173
--builtin_ntp_servers=127.0.98.20:36551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 03:28:19.701383 5113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:19.701598 5113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:19.701622 5113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:19.702791 5113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:19.702832 5113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:19.702857 5113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:28:19.704236 5113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:19.704481 5113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:19.704788 5113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:19.709574 5119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:19.709616 5118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:19.709913 5113 server_base.cc:1034] running on GCE node
W20260927 03:28:19.709957 5121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:19.710196 5113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:19.710471 5113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:19.711638 5113 hybrid_clock.cc:648] HybridClock initialized: now 1790479699711620 us; error 33 us; skew 500 ppm
I20260927 03:28:19.713025 5113 webserver.cc:460] Webserver started at http://127.0.98.1:44713/ using document root <none> and password file <none>
I20260927 03:28:19.713294 5113 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:19.713351 5113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:19.713474 5113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:19.714834 5113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-0/data/instance:
uuid: "44d85a12270e421bb237df0fdeea525d"
format_stamp: "Formatted at 2026-09-27 03:28:19 on dist-test-slave-j0hf"
server_key: "4c4f2d5b8d3312f96522cbeeb2b8a83d"
server_key_iv: "08009d68155e9eafedd5424c8ce47866"
server_key_version: "encryptionkey@0"
I20260927 03:28:19.715224 5113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-0/wal/instance:
uuid: "44d85a12270e421bb237df0fdeea525d"
format_stamp: "Formatted at 2026-09-27 03:28:19 on dist-test-slave-j0hf"
server_key: "4c4f2d5b8d3312f96522cbeeb2b8a83d"
server_key_iv: "08009d68155e9eafedd5424c8ce47866"
server_key_version: "encryptionkey@0"
I20260927 03:28:19.716851 5113 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260927 03:28:19.717706 5127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:19.717896 5113 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:19.717998 5113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "44d85a12270e421bb237df0fdeea525d"
format_stamp: "Formatted at 2026-09-27 03:28:19 on dist-test-slave-j0hf"
server_key: "4c4f2d5b8d3312f96522cbeeb2b8a83d"
server_key_iv: "08009d68155e9eafedd5424c8ce47866"
server_key_version: "encryptionkey@0"
I20260927 03:28:19.718097 5113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:19.721463 5113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:19.721791 5113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:19.721946 5113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:19.722159 5113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:19.722462 5113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:19.722517 5113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:19.722558 5113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:19.722594 5113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:19.727993 5113 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41471
I20260927 03:28:19.728063 5240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41471 every 8 connection(s)
I20260927 03:28:19.728415 5113 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:28:19.731647 5111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:19.729224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34387 (local address 127.0.98.62:34173)
0927 03:28:19.729286 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:19.729289 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:19.729541 (+ 252us) server_negotiation.cc:407] Connection header received
0927 03:28:19.730011 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:19.730013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:19.730021 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:19.730143 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:19.731345 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:19.731346 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:19.731347 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:28:19.731351 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:19.731368 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:28:19.731373 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:19.731463 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:19.731563 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 03:28:19.731579 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 03:28:19.731817 5243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:19.729216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34173 (local address 127.0.98.1:34387)
0927 03:28:19.729400 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:19.729441 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:19.729909 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:19.730206 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:19.730209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:19.730217 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:19.731187 (+ 970us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:28:19.731187 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:19.731215 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:28:19.731243 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:19.731484 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:19.731488 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:19.731491 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:28:19.731553 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 03:28:19.731611 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
I20260927 03:28:19.732332 5241 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34173
I20260927 03:28:19.732434 5241 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:19.732576 5241 heartbeater.cc:511] Master 127.0.98.62:34173 requested a full tablet report, sending...
I20260927 03:28:19.733036 5053 ts_manager.cc:194] Registered new tserver with Master: 44d85a12270e421bb237df0fdeea525d (127.0.98.1:41471)
I20260927 03:28:19.737138 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 5113
I20260927 03:28:19.737262 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:28:19.737447 392 external_mini_cluster.cc:1428] Setting key 66650771a71938d34f08e1c498928217
I20260927 03:28:19.738345 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34173
--builtin_ntp_servers=127.0.98.20:36551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 03:28:19.819132 5244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:19.819345 5244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:19.819367 5244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:19.820657 5244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:19.820698 5244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:19.820760 5244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:28:19.822208 5244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:19.822464 5244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:19.822760 5244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:28:19.827831 5244 server_base.cc:1034] running on GCE node
W20260927 03:28:19.827893 5252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:19.827816 5250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:19.827805 5249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:19.828368 5244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:19.828630 5244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:19.829814 5244 hybrid_clock.cc:648] HybridClock initialized: now 1790479699829809 us; error 39 us; skew 500 ppm
I20260927 03:28:19.831200 5244 webserver.cc:460] Webserver started at http://127.0.98.2:35863/ using document root <none> and password file <none>
I20260927 03:28:19.831529 5244 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:19.831599 5244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:19.831740 5244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:19.832904 5244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-1/data/instance:
uuid: "731c8a473ed045fc9a97f23c2dd1d9f9"
format_stamp: "Formatted at 2026-09-27 03:28:19 on dist-test-slave-j0hf"
server_key: "a758c726f7e349b90c9d23370a62f997"
server_key_iv: "2ea2b13873c064d0f033ba402b5a51d5"
server_key_version: "encryptionkey@0"
I20260927 03:28:19.833297 5244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-1/wal/instance:
uuid: "731c8a473ed045fc9a97f23c2dd1d9f9"
format_stamp: "Formatted at 2026-09-27 03:28:19 on dist-test-slave-j0hf"
server_key: "a758c726f7e349b90c9d23370a62f997"
server_key_iv: "2ea2b13873c064d0f033ba402b5a51d5"
server_key_version: "encryptionkey@0"
I20260927 03:28:19.835657 5244 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 03:28:19.836349 5258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:19.836525 5244 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:19.836627 5244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "731c8a473ed045fc9a97f23c2dd1d9f9"
format_stamp: "Formatted at 2026-09-27 03:28:19 on dist-test-slave-j0hf"
server_key: "a758c726f7e349b90c9d23370a62f997"
server_key_iv: "2ea2b13873c064d0f033ba402b5a51d5"
server_key_version: "encryptionkey@0"
I20260927 03:28:19.836711 5244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:19.840130 5244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:19.840425 5244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:19.840571 5244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:19.840777 5244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:19.841079 5244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:19.841141 5244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:19.841181 5244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:19.841207 5244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:19.847221 5244 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36855
I20260927 03:28:19.847267 5371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36855 every 8 connection(s)
I20260927 03:28:19.847653 5244 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:28:19.850983 5111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:19.848414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33339 (local address 127.0.98.62:34173)
0927 03:28:19.848484 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:19.848486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:19.848731 (+ 245us) server_negotiation.cc:407] Connection header received
0927 03:28:19.849332 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:19.849333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:19.849339 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:19.849391 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:19.850711 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:19.850713 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:19.850713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:28:19.850718 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:19.850736 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:28:19.850741 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:19.850797 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:19.850896 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 03:28:19.850914 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 03:28:19.851183 5374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:19.848405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34173 (local address 127.0.98.2:33339)
0927 03:28:19.848612 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:19.848645 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:19.849202 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:19.849466 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:19.849471 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:19.849480 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:19.850541 (+ 1061us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:28:19.850542 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:19.850571 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:28:19.850609 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:19.850792 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:19.850797 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:19.850800 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:28:19.850894 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 03:28:19.850958 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
I20260927 03:28:19.851773 5372 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34173
I20260927 03:28:19.851850 5372 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:19.852011 5372 heartbeater.cc:511] Master 127.0.98.62:34173 requested a full tablet report, sending...
I20260927 03:28:19.852382 5053 ts_manager.cc:194] Registered new tserver with Master: 731c8a473ed045fc9a97f23c2dd1d9f9 (127.0.98.2:36855)
I20260927 03:28:19.852896 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 5244
I20260927 03:28:19.852994 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:28:19.853147 392 external_mini_cluster.cc:1428] Setting key 8d72ed0cddc9639326b7091d2048d3bd
I20260927 03:28:19.853953 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34173
--builtin_ntp_servers=127.0.98.20:36551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 03:28:19.949577 5375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:19.949823 5375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:19.949860 5375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:19.951174 5375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:19.951243 5375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:19.951287 5375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:28:19.952673 5375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:19.952960 5375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:19.953262 5375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:19.958642 5380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:19.958648 5381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:19.958664 5375 server_base.cc:1034] running on GCE node
W20260927 03:28:19.958662 5383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:19.959072 5375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:19.959343 5375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:19.960525 5375 hybrid_clock.cc:648] HybridClock initialized: now 1790479699960528 us; error 47 us; skew 500 ppm
I20260927 03:28:19.961867 5375 webserver.cc:460] Webserver started at http://127.0.98.3:42079/ using document root <none> and password file <none>
I20260927 03:28:19.962133 5375 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:19.962209 5375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:19.962347 5375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:19.963528 5375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-2/data/instance:
uuid: "5171b60ef95d4507b8b371d9a2a3fabf"
format_stamp: "Formatted at 2026-09-27 03:28:19 on dist-test-slave-j0hf"
server_key: "7e8ea18023bdb47b5d2e915a4772cd5a"
server_key_iv: "1ca7f2fd483f89ebd4b80ff98b53dfaf"
server_key_version: "encryptionkey@0"
I20260927 03:28:19.963943 5375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5171b60ef95d4507b8b371d9a2a3fabf"
format_stamp: "Formatted at 2026-09-27 03:28:19 on dist-test-slave-j0hf"
server_key: "7e8ea18023bdb47b5d2e915a4772cd5a"
server_key_iv: "1ca7f2fd483f89ebd4b80ff98b53dfaf"
server_key_version: "encryptionkey@0"
I20260927 03:28:19.965317 5375 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:28:19.966017 5389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:19.966197 5375 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:28:19.966296 5375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "5171b60ef95d4507b8b371d9a2a3fabf"
format_stamp: "Formatted at 2026-09-27 03:28:19 on dist-test-slave-j0hf"
server_key: "7e8ea18023bdb47b5d2e915a4772cd5a"
server_key_iv: "1ca7f2fd483f89ebd4b80ff98b53dfaf"
server_key_version: "encryptionkey@0"
I20260927 03:28:19.966384 5375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:19.969983 5375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:19.970259 5375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:19.970453 5375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:19.970680 5375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:19.970991 5375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:19.971045 5375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:19.971084 5375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:19.971119 5375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:19.977191 5375 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39951
I20260927 03:28:19.977231 5502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39951 every 8 connection(s)
I20260927 03:28:19.977571 5375 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:28:19.978659 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 5375
I20260927 03:28:19.978760 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790479675865103-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:28:19.978973 392 external_mini_cluster.cc:1428] Setting key 54a48baa09979e517704bb706d58e770
I20260927 03:28:19.980728 5111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:19.978265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58927 (local address 127.0.98.62:34173)
0927 03:28:19.978312 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:19.978315 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:19.978616 (+ 301us) server_negotiation.cc:407] Connection header received
0927 03:28:19.979191 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:19.979193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:19.979200 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:19.979252 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:19.980514 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:19.980515 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:19.980516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:28:19.980519 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:19.980533 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:28:19.980537 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:19.980585 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:19.980664 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 03:28:19.980675 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:28:19.980971 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:19.978238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34173 (local address 127.0.98.3:58927)
0927 03:28:19.978441 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:19.978503 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:19.979089 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:19.979324 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:19.979328 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:19.979338 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:19.980378 (+ 1040us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:28:19.980378 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:19.980407 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:28:19.980436 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:19.980607 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:19.980611 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:19.980614 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:28:19.980679 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 03:28:19.980731 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":104,"threads_started":1}
I20260927 03:28:19.981591 5503 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34173
I20260927 03:28:19.981689 5503 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:19.981834 5503 heartbeater.cc:511] Master 127.0.98.62:34173 requested a full tablet report, sending...
I20260927 03:28:19.982196 5053 ts_manager.cc:194] Registered new tserver with Master: 5171b60ef95d4507b8b371d9a2a3fabf (127.0.98.3:39951)
I20260927 03:28:19.991024 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:28:19.993723 5111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:19.992523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59832 (local address 127.0.98.62:34173)
0927 03:28:19.992617 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:19.992619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:19.992719 (+ 100us) server_negotiation.cc:407] Connection header received
0927 03:28:19.992791 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:19.992792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:19.992798 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:19.992873 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:19.993459 (+ 586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:19.993460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:19.993460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:28:19.993465 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:19.993481 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:28:19.993485 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:19.993557 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:19.993631 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 03:28:19.993646 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 03:28:19.995316 5053 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:28:19.997104 5053 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:28:20.001049 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:19.999364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39951 (local address 127.0.0.1:38882)
0927 03:28:19.999543 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:19.999552 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:19.999585 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:20.000046 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:20.000051 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:20.000058 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:20.000610 (+ 552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:28:20.000611 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:20.000629 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:28:20.000655 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:20.000886 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:20.000893 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:20.000896 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:28:20.000963 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 03:28:20.000977 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":124,"threads_started":1}
I20260927 03:28:20.001168 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:19.999299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36855 (local address 127.0.0.1:58648)
0927 03:28:19.999442 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:19.999484 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:19.999543 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:20.000369 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:20.000370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:20.000373 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:20.000754 (+ 381us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:28:20.000755 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:20.000762 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:28:20.000775 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:20.001027 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:20.001028 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:20.001029 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 03:28:20.001075 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 03:28:20.001089 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":72,"threads_started":1}
I20260927 03:28:20.001421 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:19.999409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58648 (local address 127.0.98.2:36855)
0927 03:28:19.999894 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:19.999899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:19.999908 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:28:19.999932 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:19.999934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:19.999943 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:20.000018 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:20.000864 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:20.000871 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:20.000871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:28:20.000876 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:20.000910 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:28:20.000938 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:20.000999 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:20.001318 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 03:28:20.001335 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":110,"threads_started":1}
I20260927 03:28:20.001523 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:19.999691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38882 (local address 127.0.98.3:39951)
0927 03:28:19.999887 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:19.999893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:19.999903 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:28:19.999932 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:19.999934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:19.999943 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:20.000275 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:20.000733 (+ 458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:20.000739 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:20.000739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:28:20.000744 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:20.000778 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:28:20.000802 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:20.000891 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:20.001388 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 03:28:20.001403 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":102,"threads_started":1}
I20260927 03:28:20.001606 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:19.999542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41471 (local address 127.0.0.1:38498)
0927 03:28:19.999707 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:19.999716 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:19.999759 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:20.000759 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:20.000760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:20.000762 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:20.001227 (+ 465us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:28:20.001227 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:20.001233 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:28:20.001246 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:20.001469 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:20.001470 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:20.001470 (+ 0us) client_negotiation.cc:769] Sending connection context
0927 03:28:20.001528 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 03:28:20.001540 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":59,"threads_started":1}
I20260927 03:28:20.001804 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:19.999729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38498 (local address 127.0.98.1:41471)
0927 03:28:19.999918 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:19.999922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:19.999931 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:28:19.999953 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:19.999955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:19.999962 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:20.000075 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:20.001328 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:20.001334 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:20.001334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:28:20.001338 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:20.001372 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:28:20.001394 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:20.001457 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:20.001704 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 03:28:20.001720 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":81,"threads_started":1}
I20260927 03:28:20.002151 5437 tablet_service.cc:1467] Processing CreateTablet for tablet b7993e3755a6424e867929ef8469ee82 (DEFAULT_TABLE table=test-table [id=bab46e91d23842bcb5c0dfdd9cecf96c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:20.002240 5306 tablet_service.cc:1467] Processing CreateTablet for tablet b7993e3755a6424e867929ef8469ee82 (DEFAULT_TABLE table=test-table [id=bab46e91d23842bcb5c0dfdd9cecf96c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:20.002493 5306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7993e3755a6424e867929ef8469ee82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:20.002504 5437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7993e3755a6424e867929ef8469ee82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:20.004092 5175 tablet_service.cc:1467] Processing CreateTablet for tablet b7993e3755a6424e867929ef8469ee82 (DEFAULT_TABLE table=test-table [id=bab46e91d23842bcb5c0dfdd9cecf96c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:20.004437 5175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7993e3755a6424e867929ef8469ee82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:20.005270 5517 tablet_bootstrap.cc:492] T b7993e3755a6424e867929ef8469ee82 P 731c8a473ed045fc9a97f23c2dd1d9f9: Bootstrap starting.
I20260927 03:28:20.005646 5518 tablet_bootstrap.cc:492] T b7993e3755a6424e867929ef8469ee82 P 5171b60ef95d4507b8b371d9a2a3fabf: Bootstrap starting.
I20260927 03:28:20.005906 5517 tablet_bootstrap.cc:654] T b7993e3755a6424e867929ef8469ee82 P 731c8a473ed045fc9a97f23c2dd1d9f9: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:20.006125 5517 log.cc:824] T b7993e3755a6424e867929ef8469ee82 P 731c8a473ed045fc9a97f23c2dd1d9f9: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:20.006264 5518 tablet_bootstrap.cc:654] T b7993e3755a6424e867929ef8469ee82 P 5171b60ef95d4507b8b371d9a2a3fabf: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:20.006561 5518 log.cc:824] T b7993e3755a6424e867929ef8469ee82 P 5171b60ef95d4507b8b371d9a2a3fabf: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:20.006829 5517 tablet_bootstrap.cc:492] T b7993e3755a6424e867929ef8469ee82 P 731c8a473ed045fc9a97f23c2dd1d9f9: No bootstrap required, opened a new log
I20260927 03:28:20.006891 5517 ts_tablet_manager.cc:1397] T b7993e3755a6424e867929ef8469ee82 P 731c8a473ed045fc9a97f23c2dd1d9f9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:28:20.007452 5518 tablet_bootstrap.cc:492] T b7993e3755a6424e867929ef8469ee82 P 5171b60ef95d4507b8b371d9a2a3fabf: No bootstrap required, opened a new log
I20260927 03:28:20.007510 5518 ts_tablet_manager.cc:1397] T b7993e3755a6424e867929ef8469ee82 P 5171b60ef95d4507b8b371d9a2a3fabf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:28:20.007776 5521 tablet_bootstrap.cc:492] T b7993e3755a6424e867929ef8469ee82 P 44d85a12270e421bb237df0fdeea525d: Bootstrap starting.
I20260927 03:28:20.008524 5521 tablet_bootstrap.cc:654] T b7993e3755a6424e867929ef8469ee82 P 44d85a12270e421bb237df0fdeea525d: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:20.008831 5521 log.cc:824] T b7993e3755a6424e867929ef8469ee82 P 44d85a12270e421bb237df0fdeea525d: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:20.009696 5521 tablet_bootstrap.cc:492] T b7993e3755a6424e867929ef8469ee82 P 44d85a12270e421bb237df0fdeea525d: No bootstrap required, opened a new log
I20260927 03:28:20.009768 5521 ts_tablet_manager.cc:1397] T b7993e3755a6424e867929ef8469ee82 P 44d85a12270e421bb237df0fdeea525d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:28:20.009774 5518 raft_consensus.cc:348] T b7993e3755a6424e867929ef8469ee82 P 5171b60ef95d4507b8b371d9a2a3fabf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731c8a473ed045fc9a97f23c2dd1d9f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36855 } } peers { permanent_uuid: "44d85a12270e421bb237df0fdeea525d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41471 } } peers { permanent_uuid: "5171b60ef95d4507b8b371d9a2a3fabf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39951 } }
I20260927 03:28:20.009769 5517 raft_consensus.cc:348] T b7993e3755a6424e867929ef8469ee82 P 731c8a473ed045fc9a97f23c2dd1d9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731c8a473ed045fc9a97f23c2dd1d9f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36855 } } peers { permanent_uuid: "44d85a12270e421bb237df0fdeea525d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41471 } } peers { permanent_uuid: "5171b60ef95d4507b8b371d9a2a3fabf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39951 } }
I20260927 03:28:20.009955 5517 raft_consensus.cc:374] T b7993e3755a6424e867929ef8469ee82 P 731c8a473ed045fc9a97f23c2dd1d9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:20.009954 5518 raft_consensus.cc:374] T b7993e3755a6424e867929ef8469ee82 P 5171b60ef95d4507b8b371d9a2a3fabf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:20.010000 5518 raft_consensus.cc:729] T b7993e3755a6424e867929ef8469ee82 P 5171b60ef95d4507b8b371d9a2a3fabf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5171b60ef95d4507b8b371d9a2a3fabf, State: Initialized, Role: FOLLOWER
I20260927 03:28:20.010025 5517 raft_consensus.cc:729] T b7993e3755a6424e867929ef8469ee82 P 731c8a473ed045fc9a97f23c2dd1d9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731c8a473ed045fc9a97f23c2dd1d9f9, State: Initialized, Role: FOLLOWER
I20260927 03:28:20.010152 5518 consensus_queue.cc:260] T b7993e3755a6424e867929ef8469ee82 P 5171b60ef95d4507b8b371d9a2a3fabf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731c8a473ed045fc9a97f23c2dd1d9f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36855 } } peers { permanent_uuid: "44d85a12270e421bb237df0fdeea525d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41471 } } peers { permanent_uuid: "5171b60ef95d4507b8b371d9a2a3fabf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39951 } }
I20260927 03:28:20.010197 5517 consensus_queue.cc:260] T b7993e3755a6424e867929ef8469ee82 P 731c8a473ed045fc9a97f23c2dd1d9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731c8a473ed045fc9a97f23c2dd1d9f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36855 } } peers { permanent_uuid: "44d85a12270e421bb237df0fdeea525d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41471 } } peers { permanent_uuid: "5171b60ef95d4507b8b371d9a2a3fabf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39951 } }
I20260927 03:28:20.010424 5518 ts_tablet_manager.cc:1428] T b7993e3755a6424e867929ef8469ee82 P 5171b60ef95d4507b8b371d9a2a3fabf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:28:20.010475 5517 ts_tablet_manager.cc:1428] T b7993e3755a6424e867929ef8469ee82 P 731c8a473ed045fc9a97f23c2dd1d9f9: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260927 03:28:20.010519 5372 heartbeater.cc:503] Master 127.0.98.62:34173 was elected leader, sending a full tablet report...
I20260927 03:28:20.010905 5503 heartbeater.cc:503] Master 127.0.98.62:34173 was elected leader, sending a full tablet report...
I20260927 03:28:20.012665 5521 raft_consensus.cc:348] T b7993e3755a6424e867929ef8469ee82 P 44d85a12270e421bb237df0fdeea525d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731c8a473ed045fc9a97f23c2dd1d9f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36855 } } peers { permanent_uuid: "44d85a12270e421bb237df0fdeea525d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41471 } } peers { permanent_uuid: "5171b60ef95d4507b8b371d9a2a3fabf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39951 } }
I20260927 03:28:20.012794 5521 raft_consensus.cc:374] T b7993e3755a6424e867929ef8469ee82 P 44d85a12270e421bb237df0fdeea525d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:20.012816 5521 raft_consensus.cc:729] T b7993e3755a6424e867929ef8469ee82 P 44d85a12270e421bb237df0fdeea525d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d85a12270e421bb237df0fdeea525d, State: Initialized, Role: FOLLOWER
I20260927 03:28:20.012962 5521 consensus_queue.cc:260] T b7993e3755a6424e867929ef8469ee82 P 44d85a12270e421bb237df0fdeea525d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731c8a473ed045fc9a97f23c2dd1d9f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36855 } } peers { permanent_uuid: "44d85a12270e421bb237df0fdeea525d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41471 } } peers { permanent_uuid: "5171b60ef95d4507b8b371d9a2a3fabf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39951 } }
I20260927 03:28:20.013222 5521 ts_tablet_manager.cc:1428] T b7993e3755a6424e867929ef8469ee82 P 44d85a12270e421bb237df0fdeea525d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:28:20.013372 5523 raft_consensus.cc:482] T b7993e3755a6424e867929ef8469ee82 P 5171b60ef95d4507b8b371d9a2a3fabf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:28:20.013435 5523 raft_consensus.cc:504] T b7993e3755a6424e867929ef8469ee82 P 5171b60ef95d4507b8b371d9a2a3fabf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731c8a473ed045fc9a97f23c2dd1d9f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36855 } } peers { permanent_uuid: "44d85a12270e421bb237df0fdeea525d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41471 } } peers { permanent_uuid: "5171b60ef95d4507b8b371d9a2a3fabf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39951 } }
I20260927 03:28:20.013540 5241 heartbeater.cc:503] Master 127.0.98.62:34173 was elected leader, sending a full tablet report...
I20260927 03:28:20.013769 5523 leader_election.cc:288] T b7993e3755a6424e867929ef8469ee82 P 5171b60ef95d4507b8b371d9a2a3fabf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 731c8a473ed045fc9a97f23c2dd1d9f9 (127.0.98.2:36855), 44d85a12270e421bb237df0fdeea525d (127.0.98.1:41471)
I20260927 03:28:20.015226 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:20.013957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36855 (local address 127.0.98.3:33445)
0927 03:28:20.014198 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:20.014208 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:20.014251 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:20.014364 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:20.014365 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:20.014368 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:20.014863 (+ 495us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:28:20.014863 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:20.014871 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:28:20.014884 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:20.015065 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:20.015066 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:20.015067 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 03:28:20.015146 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 03:28:20.015161 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227}
I20260927 03:28:20.015226 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:20.014008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33445 (local address 127.0.98.2:36855)
0927 03:28:20.014064 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:20.014066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:20.014275 (+ 209us) server_negotiation.cc:407] Connection header received
0927 03:28:20.014301 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:20.014302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:20.014308 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:20.014361 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:20.014965 (+ 604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:20.014966 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:20.014967 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:28:20.014970 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:20.014985 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:28:20.014995 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:20.015067 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:20.015150 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 03:28:20.015164 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 03:28:20.015486 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:20.014152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58615 (local address 127.0.98.1:41471)
0927 03:28:20.014194 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:20.014196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:20.014264 (+ 68us) server_negotiation.cc:407] Connection header received
0927 03:28:20.014325 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:20.014325 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:20.014329 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:20.014382 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:20.015242 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:20.015243 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:20.015244 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:28:20.015248 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:20.015263 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:28:20.015267 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:20.015298 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:20.015390 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 03:28:20.015404 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:28:20.015514 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:20.014077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41471 (local address 127.0.98.3:58615)
0927 03:28:20.014205 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:20.014216 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:20.014273 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:20.014385 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:20.014387 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:20.014389 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:20.015133 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:28:20.015134 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:20.015141 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:28:20.015151 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:20.015343 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:20.015343 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:20.015344 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 03:28:20.015389 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 03:28:20.015403 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260927 03:28:20.015756 5326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7993e3755a6424e867929ef8469ee82" candidate_uuid: "5171b60ef95d4507b8b371d9a2a3fabf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "731c8a473ed045fc9a97f23c2dd1d9f9" is_pre_election: true
I20260927 03:28:20.015756 5177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7993e3755a6424e867929ef8469ee82" candidate_uuid: "5171b60ef95d4507b8b371d9a2a3fabf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d85a12270e421bb237df0fdeea525d" is_pre_election: true
I20260927 03:28:20.015879 5326 raft_consensus.cc:2454] T b7993e3755a6424e867929ef8469ee82 P 731c8a473ed045fc9a97f23c2dd1d9f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5171b60ef95d4507b8b371d9a2a3fabf in term 0.
I20260927 03:28:20.015879 5177 raft_consensus.cc:2454] T b7993e3755a6424e867929ef8469ee82 P 44d85a12270e421bb237df0fdeea525d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5171b60ef95d4507b8b371d9a2a3fabf in term 0.
I20260927 03:28:20.016105 5392 leader_election.cc:302] T b7993e3755a6424e867929ef8469ee82 P 5171b60ef95d4507b8b371d9a2a3fabf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5171b60ef95d4507b8b371d9a2a3fabf, 731c8a473ed045fc9a97f23c2dd1d9f9; no voters:
I20260927 03:28:20.016232 5523 raft_consensus.cc:2781] T b7993e3755a6424e867929ef8469ee82 P 5171b60ef95d4507b8b371d9a2a3fabf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:28:20.016310 5523 raft_consensus.cc:482] T b7993e3755a6424e867929ef8469ee82 P 5171b60ef95d4507b8b371d9a2a3fabf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:28:20.016342 5523 raft_consensus.cc:3037] T b7993e3755a6424e867929ef8469ee82 P 5171b60ef95d4507b8b371d9a2a3fabf [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:20.016992 5523 raft_consensus.cc:504] T b7993e3755a6424e867929ef8469ee82 P 5171b60ef95d4507b8b371d9a2a3fabf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731c8a473ed045fc9a97f23c2dd1d9f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36855 } } peers { permanent_uuid: "44d85a12270e421bb237df0fdeea525d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41471 } } peers { permanent_uuid: "5171b60ef95d4507b8b371d9a2a3fabf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39951 } }
I20260927 03:28:20.017129 5523 leader_election.cc:288] T b7993e3755a6424e867929ef8469ee82 P 5171b60ef95d4507b8b371d9a2a3fabf [CANDIDATE]: Term 1 election: Requested vote from peers 731c8a473ed045fc9a97f23c2dd1d9f9 (127.0.98.2:36855), 44d85a12270e421bb237df0fdeea525d (127.0.98.1:41471)
I20260927 03:28:20.017323 5177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7993e3755a6424e867929ef8469ee82" candidate_uuid: "5171b60ef95d4507b8b371d9a2a3fabf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d85a12270e421bb237df0fdeea525d"
I20260927 03:28:20.017289 5326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7993e3755a6424e867929ef8469ee82" candidate_uuid: "5171b60ef95d4507b8b371d9a2a3fabf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "731c8a473ed045fc9a97f23c2dd1d9f9"
I20260927 03:28:20.017416 5177 raft_consensus.cc:3037] T b7993e3755a6424e867929ef8469ee82 P 44d85a12270e421bb237df0fdeea525d [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:20.017460 5326 raft_consensus.cc:3037] T b7993e3755a6424e867929ef8469ee82 P 731c8a473ed045fc9a97f23c2dd1d9f9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:20.018056 5177 raft_consensus.cc:2454] T b7993e3755a6424e867929ef8469ee82 P 44d85a12270e421bb237df0fdeea525d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5171b60ef95d4507b8b371d9a2a3fabf in term 1.
I20260927 03:28:20.018230 5392 leader_election.cc:302] T b7993e3755a6424e867929ef8469ee82 P 5171b60ef95d4507b8b371d9a2a3fabf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d85a12270e421bb237df0fdeea525d, 5171b60ef95d4507b8b371d9a2a3fabf; no voters:
I20260927 03:28:20.018313 5523 raft_consensus.cc:2781] T b7993e3755a6424e867929ef8469ee82 P 5171b60ef95d4507b8b371d9a2a3fabf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:20.018406 5326 raft_consensus.cc:2454] T b7993e3755a6424e867929ef8469ee82 P 731c8a473ed045fc9a97f23c2dd1d9f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5171b60ef95d4507b8b371d9a2a3fabf in term 1.
I20260927 03:28:20.018469 5523 raft_consensus.cc:686] T b7993e3755a6424e867929ef8469ee82 P 5171b60ef95d4507b8b371d9a2a3fabf [term 1 LEADER]: Becoming Leader. State: Replica: 5171b60ef95d4507b8b371d9a2a3fabf, State: Running, Role: LEADER
I20260927 03:28:20.018555 5523 consensus_queue.cc:237] T b7993e3755a6424e867929ef8469ee82 P 5171b60ef95d4507b8b371d9a2a3fabf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731c8a473ed045fc9a97f23c2dd1d9f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36855 } } peers { permanent_uuid: "44d85a12270e421bb237df0fdeea525d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41471 } } peers { permanent_uuid: "5171b60ef95d4507b8b371d9a2a3fabf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39951 } }
I20260927 03:28:20.019405 5053 catalog_manager.cc:5495] T b7993e3755a6424e867929ef8469ee82 P 5171b60ef95d4507b8b371d9a2a3fabf reported cstate change: term changed from 0 to 1, leader changed from <none> to 5171b60ef95d4507b8b371d9a2a3fabf (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5171b60ef95d4507b8b371d9a2a3fabf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731c8a473ed045fc9a97f23c2dd1d9f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d85a12270e421bb237df0fdeea525d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5171b60ef95d4507b8b371d9a2a3fabf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39951 } health_report { overall_health: HEALTHY } } }
I20260927 03:28:20.029639 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:20.028602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38884 (local address 127.0.98.3:39951)
0927 03:28:20.028653 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:20.028655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:20.028665 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:28:20.028721 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:20.028723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:20.028734 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:20.028809 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:20.029352 (+ 543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:20.029353 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:20.029354 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:28:20.029358 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:20.029373 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:28:20.029378 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:20.029447 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:20.029542 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 03:28:20.029555 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 03:28:20.031397 5177 raft_consensus.cc:1261] T b7993e3755a6424e867929ef8469ee82 P 44d85a12270e421bb237df0fdeea525d [term 1 FOLLOWER]: Refusing update from remote peer 5171b60ef95d4507b8b371d9a2a3fabf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:20.031407 5326 raft_consensus.cc:1261] T b7993e3755a6424e867929ef8469ee82 P 731c8a473ed045fc9a97f23c2dd1d9f9 [term 1 FOLLOWER]: Refusing update from remote peer 5171b60ef95d4507b8b371d9a2a3fabf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:20.031826 5523 consensus_queue.cc:1035] T b7993e3755a6424e867929ef8469ee82 P 5171b60ef95d4507b8b371d9a2a3fabf [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d85a12270e421bb237df0fdeea525d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:20.031983 5527 consensus_queue.cc:1035] T b7993e3755a6424e867929ef8469ee82 P 5171b60ef95d4507b8b371d9a2a3fabf [LEADER]: Connected to new peer: Peer: permanent_uuid: "731c8a473ed045fc9a97f23c2dd1d9f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:20.034442 5530 mvcc.cc:205] Tried to move back new op lower bound from 7333804851326054400 to 7333804851276943360. Current Snapshot: MvccSnapshot[applied={T|T < 7333804851326054400}]
I20260927 03:28:20.035305 5531 mvcc.cc:205] Tried to move back new op lower bound from 7333804851326054400 to 7333804851276943360. Current Snapshot: MvccSnapshot[applied={T|T < 7333804851326054400}]
I20260927 03:28:20.036888 5053 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:28:20.036989 5053 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:28:20.037755 5053 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 401a21beba23407e90db6ee2329a4511: Sending DeleteTablet for 3 replicas of tablet b7993e3755a6424e867929ef8469ee82
I20260927 03:28:20.038070 5437 tablet_service.cc:1514] Processing DeleteTablet for tablet b7993e3755a6424e867929ef8469ee82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:20 UTC) from {username='slave'} at 127.0.0.1:38882
I20260927 03:28:20.038095 5175 tablet_service.cc:1514] Processing DeleteTablet for tablet b7993e3755a6424e867929ef8469ee82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:20 UTC) from {username='slave'} at 127.0.0.1:38498
I20260927 03:28:20.038314 5306 tablet_service.cc:1514] Processing DeleteTablet for tablet b7993e3755a6424e867929ef8469ee82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:20 UTC) from {username='slave'} at 127.0.0.1:58648
I20260927 03:28:20.038523 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 5113
I20260927 03:28:20.038532 5542 tablet_replica.cc:330] T b7993e3755a6424e867929ef8469ee82 P 731c8a473ed045fc9a97f23c2dd1d9f9: stopping tablet replica
I20260927 03:28:20.038586 5541 tablet_replica.cc:330] T b7993e3755a6424e867929ef8469ee82 P 44d85a12270e421bb237df0fdeea525d: stopping tablet replica
I20260927 03:28:20.038636 5542 raft_consensus.cc:2229] T b7993e3755a6424e867929ef8469ee82 P 731c8a473ed045fc9a97f23c2dd1d9f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:28:20.038679 5541 raft_consensus.cc:2229] T b7993e3755a6424e867929ef8469ee82 P 44d85a12270e421bb237df0fdeea525d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:28:20.038736 5542 raft_consensus.cc:2258] T b7993e3755a6424e867929ef8469ee82 P 731c8a473ed045fc9a97f23c2dd1d9f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:20.039469 5542 ts_tablet_manager.cc:1905] T b7993e3755a6424e867929ef8469ee82 P 731c8a473ed045fc9a97f23c2dd1d9f9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 03:28:20.044148 5039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41471 (error 108)
I20260927 03:28:20.044340 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 5244
I20260927 03:28:20.045821 5540 tablet_replica.cc:330] T b7993e3755a6424e867929ef8469ee82 P 5171b60ef95d4507b8b371d9a2a3fabf: stopping tablet replica
I20260927 03:28:20.045960 5540 raft_consensus.cc:2229] T b7993e3755a6424e867929ef8469ee82 P 5171b60ef95d4507b8b371d9a2a3fabf [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:28:20.046104 5540 raft_consensus.cc:2258] T b7993e3755a6424e867929ef8469ee82 P 5171b60ef95d4507b8b371d9a2a3fabf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:20.046578 5540 ts_tablet_manager.cc:1905] T b7993e3755a6424e867929ef8469ee82 P 5171b60ef95d4507b8b371d9a2a3fabf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:20.046626 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:20.046464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41471 (local address 127.0.0.1:38510)
0927 03:28:20.046531 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:20.046587 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 03:28:20.046741 5039 catalog_manager.cc:4584] TS 44d85a12270e421bb237df0fdeea525d (127.0.98.1:41471): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b7993e3755a6424e867929ef8469ee82: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41471: connect: Connection refused (error 111)
I20260927 03:28:20.099043 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 5375
I20260927 03:28:20.099170 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:20.099037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41471 (local address 127.0.0.1:38522)
0927 03:28:20.099098 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:20.099127 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 03:28:20.101009 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:20.100881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36855 (local address 127.0.0.1:58652)
0927 03:28:20.100937 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:20.100973 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 03:28:20.101132 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:20.100988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41471 (local address 127.0.0.1:38524)
0927 03:28:20.101054 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:20.101092 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 03:28:20.104301 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:20.104166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39951 (local address 127.0.0.1:38892)
0927 03:28:20.104237 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:20.104268 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 03:28:20.104295 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 5022
2026-09-27T03:28:20Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (672 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 03:28:20.124266 392 mini_oidc.cc:152] Starting JWKS server
I20260927 03:28:20.125319 392 webserver.cc:460] Webserver started at https://127.0.0.1:38911/ using document root <none> and password file <none>
I20260927 03:28:20.125422 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 03:28:20.125597 392 webserver.cc:460] Webserver started at http://127.0.0.1:36559/ using document root <none> and password file <none>
2026-09-27T03:28:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:28:20Z Disabled control of system clock
I20260927 03:28:20.133435 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43431
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43431
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38911/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479675865103-392-0/testchainca.cert with env {}
W20260927 03:28:20.215859 5549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:20.216061 5549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:20.216090 5549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:20.217185 5549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:28:20.217217 5549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:20.217231 5549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:20.217242 5549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:28:20.217254 5549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 03:28:20.217279 5549 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 03:28:20.217290 5549 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38911/jwks.json
I20260927 03:28:20.218720 5549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43431
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38911/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479675865103-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:20.218909 5549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:20.219066 5549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:20.223686 5554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:20.223711 5555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:20.223737 5549 server_base.cc:1034] running on GCE node
W20260927 03:28:20.223707 5557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:20.224135 5549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:20.224419 5549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:20.225580 5549 hybrid_clock.cc:648] HybridClock initialized: now 1790479700225558 us; error 38 us; skew 500 ppm
I20260927 03:28:20.235172 5549 webserver.cc:460] Webserver started at http://127.0.98.62:33343/ using document root <none> and password file <none>
I20260927 03:28:20.235539 5549 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:20.235618 5549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:20.235785 5549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:20.236901 5549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "12f35617a9944b36a7a1183550135df4"
format_stamp: "Formatted at 2026-09-27 03:28:20 on dist-test-slave-j0hf"
server_key: "99b27e2a5f276f159cd92b5b92eafcf9"
server_key_iv: "b8971aff24feac0bd78a84b036e4da9a"
server_key_version: "encryptionkey@0"
I20260927 03:28:20.237349 5549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "12f35617a9944b36a7a1183550135df4"
format_stamp: "Formatted at 2026-09-27 03:28:20 on dist-test-slave-j0hf"
server_key: "99b27e2a5f276f159cd92b5b92eafcf9"
server_key_iv: "b8971aff24feac0bd78a84b036e4da9a"
server_key_version: "encryptionkey@0"
I20260927 03:28:20.238653 5549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:28:20.239529 5563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:20.239714 5549 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:20.239831 5549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "12f35617a9944b36a7a1183550135df4"
format_stamp: "Formatted at 2026-09-27 03:28:20 on dist-test-slave-j0hf"
server_key: "99b27e2a5f276f159cd92b5b92eafcf9"
server_key_iv: "b8971aff24feac0bd78a84b036e4da9a"
server_key_version: "encryptionkey@0"
I20260927 03:28:20.239923 5549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:20.262086 5549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:20.262463 5549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:20.262635 5549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:20.267066 5549 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43431
I20260927 03:28:20.267100 5615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43431 every 8 connection(s)
I20260927 03:28:20.267542 5549 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:28:20.267921 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 5549
I20260927 03:28:20.268036 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:28:20.268241 392 external_mini_cluster.cc:1428] Setting key b3985400750d453fb6f30171b8c0d6d3
I20260927 03:28:20.268802 5616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:20.272284 5616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12f35617a9944b36a7a1183550135df4: Bootstrap starting.
I20260927 03:28:20.273131 5616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12f35617a9944b36a7a1183550135df4: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:20.273450 5616 log.cc:824] T 00000000000000000000000000000000 P 12f35617a9944b36a7a1183550135df4: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:20.274406 5616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12f35617a9944b36a7a1183550135df4: No bootstrap required, opened a new log
I20260927 03:28:20.274909 5619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:20.268833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40004 (local address 127.0.98.62:43431)
0927 03:28:20.269049 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:20.269057 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:20.269102 (+ 45us) server_negotiation.cc:407] Connection header received
0927 03:28:20.269667 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:20.269678 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:20.269774 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:20.269949 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:20.270212 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:20.271602 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:20.272553 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:20.273181 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:20.273327 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:20.274109 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:20.274114 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:20.274114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:28:20.274325 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:20.274370 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:28:20.274386 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:20.274460 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:20.274622 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 03:28:20.274717 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":87,"threads_started":1}
I20260927 03:28:20.275997 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:28:20.276922 5616 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12f35617a9944b36a7a1183550135df4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f35617a9944b36a7a1183550135df4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43431 } }
I20260927 03:28:20.277119 5616 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12f35617a9944b36a7a1183550135df4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:20.277168 5616 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12f35617a9944b36a7a1183550135df4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12f35617a9944b36a7a1183550135df4, State: Initialized, Role: FOLLOWER
I20260927 03:28:20.277308 5616 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12f35617a9944b36a7a1183550135df4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f35617a9944b36a7a1183550135df4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43431 } }
I20260927 03:28:20.277386 5616 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12f35617a9944b36a7a1183550135df4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:28:20.277410 5616 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12f35617a9944b36a7a1183550135df4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:28:20.277443 5616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12f35617a9944b36a7a1183550135df4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:20.278002 5616 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12f35617a9944b36a7a1183550135df4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f35617a9944b36a7a1183550135df4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43431 } }
I20260927 03:28:20.278090 5616 leader_election.cc:302] T 00000000000000000000000000000000 P 12f35617a9944b36a7a1183550135df4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12f35617a9944b36a7a1183550135df4; no voters:
I20260927 03:28:20.278223 5616 leader_election.cc:288] T 00000000000000000000000000000000 P 12f35617a9944b36a7a1183550135df4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:28:20.278291 5621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12f35617a9944b36a7a1183550135df4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:20.278465 5621 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12f35617a9944b36a7a1183550135df4 [term 1 LEADER]: Becoming Leader. State: Replica: 12f35617a9944b36a7a1183550135df4, State: Running, Role: LEADER
I20260927 03:28:20.278514 5616 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12f35617a9944b36a7a1183550135df4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:28:20.278554 5621 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12f35617a9944b36a7a1183550135df4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f35617a9944b36a7a1183550135df4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43431 } }
I20260927 03:28:20.278927 5622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12f35617a9944b36a7a1183550135df4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12f35617a9944b36a7a1183550135df4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f35617a9944b36a7a1183550135df4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43431 } } }
I20260927 03:28:20.278937 5623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12f35617a9944b36a7a1183550135df4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12f35617a9944b36a7a1183550135df4. Latest consensus state: current_term: 1 leader_uuid: "12f35617a9944b36a7a1183550135df4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f35617a9944b36a7a1183550135df4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43431 } } }
I20260927 03:28:20.279062 5623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12f35617a9944b36a7a1183550135df4 [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:20.279255 5622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12f35617a9944b36a7a1183550135df4 [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:20.279346 5629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:28:20.279904 5629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:28:20.281567 5629 catalog_manager.cc:1347] Generated new cluster ID: 24e608a24d0f4f81af64e27cd17f48e4
I20260927 03:28:20.281628 5629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:28:20.286794 5629 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:28:20.287318 5629 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:28:20.290395 5629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12f35617a9944b36a7a1183550135df4: Generated new TSK 0
I20260927 03:28:20.290567 5629 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:28:20.320628 5619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:20.296941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40018 (local address 127.0.98.62:43431)
0927 03:28:20.296982 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:20.296985 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:20.297141 (+ 156us) server_negotiation.cc:407] Connection header received
0927 03:28:20.297235 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:20.297236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:20.297246 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:20.297306 (+ 60us) server_negotiation.cc:226] Negotiated authn=JWT
0927 03:28:20.297470 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:20.297928 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:20.299304 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:20.299548 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:20.299609 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:20.299835 (+ 226us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 03:28:20.320363 (+ 20528us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38911/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 03:28:20.320533 (+ 170us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40018: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38911/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 03:28:20.320982 5619 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40018: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38911/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 03:28:20.321141 5645 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:20.296879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43431 (local address 127.0.0.1:40018)
0927 03:28:20.297060 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:20.297071 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:20.297148 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:20.297335 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:20.297335 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:20.297338 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0927 03:28:20.297405 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:20.297408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:20.298243 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:20.298244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:20.299220 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:20.299223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:20.299623 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:20.299626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:20.299694 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:20.299719 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 03:28:20.320536 (+ 20817us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38911/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 03:28:20.320724 (+ 188us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43431: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38911/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:43431: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38911/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 03:28:20.321609 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 5549
E20260927 03:28:20.322432 5646 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 03:28:20.322506 5646 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T03:28:20Z chronyd exiting
I20260927 03:28:20.358908 392 test_util.cc:175] -----------------------------------------------
I20260927 03:28:20.358999 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790479675865103-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (235 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 03:28:20.359855 392 mini_oidc.cc:152] Starting JWKS server
I20260927 03:28:20.360489 392 webserver.cc:460] Webserver started at https://127.0.0.1:34315/ using document root <none> and password file <none>
I20260927 03:28:20.360559 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 03:28:20.360728 392 webserver.cc:460] Webserver started at http://127.0.0.1:42817/ using document root <none> and password file <none>
2026-09-27T03:28:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:28:20Z Disabled control of system clock
I20260927 03:28:20.368252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45691
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45691
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34315/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479675865103-392-0/testchainca.cert with env {}
W20260927 03:28:20.450030 5652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:20.450245 5652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:20.450268 5652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:20.451380 5652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:28:20.451463 5652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:20.451501 5652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:20.451516 5652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:28:20.451529 5652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 03:28:20.451540 5652 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 03:28:20.451552 5652 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34315/jwks.json
I20260927 03:28:20.453087 5652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45691
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34315/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479675865103-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:20.453281 5652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:20.453446 5652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:20.458334 5658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:20.458308 5657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:20.458330 5660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:20.458593 5652 server_base.cc:1034] running on GCE node
I20260927 03:28:20.458847 5652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:20.459168 5652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:20.460355 5652 hybrid_clock.cc:648] HybridClock initialized: now 1790479700460339 us; error 50 us; skew 500 ppm
I20260927 03:28:20.461563 5652 webserver.cc:460] Webserver started at http://127.0.98.62:45833/ using document root <none> and password file <none>
I20260927 03:28:20.461815 5652 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:20.461862 5652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:20.462013 5652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:20.463109 5652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "6419a84f9bab4712a8502197ae5f0ee0"
format_stamp: "Formatted at 2026-09-27 03:28:20 on dist-test-slave-j0hf"
server_key: "d848031b5e93cc8cc0ec974d78c7c2a7"
server_key_iv: "3af77d435897b621ad1937faa2dc95c1"
server_key_version: "encryptionkey@0"
I20260927 03:28:20.463542 5652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "6419a84f9bab4712a8502197ae5f0ee0"
format_stamp: "Formatted at 2026-09-27 03:28:20 on dist-test-slave-j0hf"
server_key: "d848031b5e93cc8cc0ec974d78c7c2a7"
server_key_iv: "3af77d435897b621ad1937faa2dc95c1"
server_key_version: "encryptionkey@0"
I20260927 03:28:20.464906 5652 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:20.465708 5666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:20.465902 5652 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:20.465981 5652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "6419a84f9bab4712a8502197ae5f0ee0"
format_stamp: "Formatted at 2026-09-27 03:28:20 on dist-test-slave-j0hf"
server_key: "d848031b5e93cc8cc0ec974d78c7c2a7"
server_key_iv: "3af77d435897b621ad1937faa2dc95c1"
server_key_version: "encryptionkey@0"
I20260927 03:28:20.466075 5652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:20.514809 5652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:20.515187 5652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:20.515352 5652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:20.519181 5652 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45691
I20260927 03:28:20.519243 5718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45691 every 8 connection(s)
I20260927 03:28:20.519626 5652 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:28:20.520562 5719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:20.522840 5719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6419a84f9bab4712a8502197ae5f0ee0: Bootstrap starting.
I20260927 03:28:20.523154 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 5652
I20260927 03:28:20.523283 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:28:20.523494 392 external_mini_cluster.cc:1428] Setting key f262293174b9e6a6eac6bd6752ede88d
I20260927 03:28:20.523588 5719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6419a84f9bab4712a8502197ae5f0ee0: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:20.523919 5719 log.cc:824] T 00000000000000000000000000000000 P 6419a84f9bab4712a8502197ae5f0ee0: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:20.524644 5719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6419a84f9bab4712a8502197ae5f0ee0: No bootstrap required, opened a new log
I20260927 03:28:20.526791 5719 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6419a84f9bab4712a8502197ae5f0ee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6419a84f9bab4712a8502197ae5f0ee0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45691 } }
I20260927 03:28:20.526954 5719 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6419a84f9bab4712a8502197ae5f0ee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:20.526985 5719 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6419a84f9bab4712a8502197ae5f0ee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6419a84f9bab4712a8502197ae5f0ee0, State: Initialized, Role: FOLLOWER
I20260927 03:28:20.527102 5719 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6419a84f9bab4712a8502197ae5f0ee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6419a84f9bab4712a8502197ae5f0ee0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45691 } }
I20260927 03:28:20.527172 5719 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6419a84f9bab4712a8502197ae5f0ee0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:28:20.527200 5719 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6419a84f9bab4712a8502197ae5f0ee0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:28:20.527251 5719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6419a84f9bab4712a8502197ae5f0ee0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:20.527973 5719 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6419a84f9bab4712a8502197ae5f0ee0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6419a84f9bab4712a8502197ae5f0ee0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45691 } }
I20260927 03:28:20.528081 5719 leader_election.cc:302] T 00000000000000000000000000000000 P 6419a84f9bab4712a8502197ae5f0ee0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6419a84f9bab4712a8502197ae5f0ee0; no voters:
I20260927 03:28:20.528234 5719 leader_election.cc:288] T 00000000000000000000000000000000 P 6419a84f9bab4712a8502197ae5f0ee0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:28:20.528273 5724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6419a84f9bab4712a8502197ae5f0ee0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:20.528406 5724 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6419a84f9bab4712a8502197ae5f0ee0 [term 1 LEADER]: Becoming Leader. State: Replica: 6419a84f9bab4712a8502197ae5f0ee0, State: Running, Role: LEADER
I20260927 03:28:20.528504 5724 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6419a84f9bab4712a8502197ae5f0ee0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6419a84f9bab4712a8502197ae5f0ee0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45691 } }
I20260927 03:28:20.528512 5719 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6419a84f9bab4712a8502197ae5f0ee0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:28:20.528944 5725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6419a84f9bab4712a8502197ae5f0ee0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6419a84f9bab4712a8502197ae5f0ee0. Latest consensus state: current_term: 1 leader_uuid: "6419a84f9bab4712a8502197ae5f0ee0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6419a84f9bab4712a8502197ae5f0ee0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45691 } } }
I20260927 03:28:20.528950 5726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6419a84f9bab4712a8502197ae5f0ee0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6419a84f9bab4712a8502197ae5f0ee0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6419a84f9bab4712a8502197ae5f0ee0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45691 } } }
I20260927 03:28:20.529031 5725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6419a84f9bab4712a8502197ae5f0ee0 [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:20.529047 5726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6419a84f9bab4712a8502197ae5f0ee0 [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:20.529251 5728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:28:20.529768 5728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:28:20.530611 5723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:20.524027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55052 (local address 127.0.98.62:45691)
0927 03:28:20.524290 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:20.524298 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:20.524320 (+ 22us) server_negotiation.cc:407] Connection header received
0927 03:28:20.524809 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:20.524817 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:20.524876 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:20.524994 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:20.525255 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:20.526113 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:20.527307 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:20.527951 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:20.528087 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:20.529689 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:20.529694 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:20.529694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:28:20.529898 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:20.529952 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:28:20.529966 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:20.530301 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:20.530335 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 03:28:20.530412 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":95,"threads_started":1}
I20260927 03:28:20.531386 5728 catalog_manager.cc:1347] Generated new cluster ID: 9cb1566542244b2da3186c5b5fa4f9ea
I20260927 03:28:20.531461 5728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:28:20.531571 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:28:20.559993 5728 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:28:20.560539 5728 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:28:20.572899 5728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6419a84f9bab4712a8502197ae5f0ee0: Generated new TSK 0
I20260927 03:28:20.573218 5728 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 03:28:20.602622 5749 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 03:28:20.602722 5749 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 03:28:20.603049 5723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:20.588702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55066 (local address 127.0.98.62:45691)
0927 03:28:20.588751 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:20.588753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:20.588853 (+ 100us) server_negotiation.cc:407] Connection header received
0927 03:28:20.588909 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:20.588910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:20.588923 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:20.589006 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0927 03:28:20.589133 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:20.589869 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:20.590684 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:20.590967 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:20.591040 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:20.591191 (+ 151us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 03:28:20.602808 (+ 11617us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34315/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 03:28:20.602976 (+ 168us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55066: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34315/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":27}
W20260927 03:28:20.603132 5723 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55066: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34315/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 03:28:20.603107 5748 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:20.588629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45691 (local address 127.0.0.1:55066)
0927 03:28:20.588774 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:20.588784 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:20.588851 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:20.589010 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:20.589011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:20.589013 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 03:28:20.589062 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:20.589064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:20.589950 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:20.589950 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:20.590596 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:20.590597 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:20.591044 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:20.591047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:20.591085 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:20.591089 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 03:28:20.602935 (+ 11846us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34315/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 03:28:20.603032 (+ 97us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45691: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34315/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":97,"threads_started":1}
I20260927 03:28:20.603453 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 5652
2026-09-27T03:28:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (281 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 03:28:20.641309 392 mini_oidc.cc:152] Starting JWKS server
I20260927 03:28:20.642016 392 webserver.cc:460] Webserver started at https://127.0.0.1:36493/ using document root <none> and password file <none>
I20260927 03:28:20.642097 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 03:28:20.642302 392 webserver.cc:460] Webserver started at http://127.0.0.1:36797/ using document root <none> and password file <none>
2026-09-27T03:28:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:28:20Z Disabled control of system clock
I20260927 03:28:20.649773 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34729
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34729
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36493/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 03:28:20.729836 5755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:20.730106 5755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:20.730144 5755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:20.731372 5755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:28:20.731462 5755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:20.731489 5755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:20.731518 5755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:28:20.731544 5755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 03:28:20.731571 5755 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 03:28:20.731594 5755 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36493/jwks.json
I20260927 03:28:20.733090 5755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34729
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36493/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:20.733383 5755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:20.733706 5755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:20.738476 5763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:20.738462 5761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:20.738480 5760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:20.738675 5755 server_base.cc:1034] running on GCE node
I20260927 03:28:20.738973 5755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:20.739264 5755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:20.740429 5755 hybrid_clock.cc:648] HybridClock initialized: now 1790479700740413 us; error 47 us; skew 500 ppm
I20260927 03:28:20.741608 5755 webserver.cc:460] Webserver started at http://127.0.98.62:34115/ using document root <none> and password file <none>
I20260927 03:28:20.741856 5755 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:20.741926 5755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:20.742075 5755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:20.743122 5755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "1170a754e75f4fc9b538abe78b483eae"
format_stamp: "Formatted at 2026-09-27 03:28:20 on dist-test-slave-j0hf"
server_key: "3b274a59cc7940dbe37585dd436b4580"
server_key_iv: "e127c152ae37b154f7c26f789dfaf66b"
server_key_version: "encryptionkey@0"
I20260927 03:28:20.743543 5755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "1170a754e75f4fc9b538abe78b483eae"
format_stamp: "Formatted at 2026-09-27 03:28:20 on dist-test-slave-j0hf"
server_key: "3b274a59cc7940dbe37585dd436b4580"
server_key_iv: "e127c152ae37b154f7c26f789dfaf66b"
server_key_version: "encryptionkey@0"
I20260927 03:28:20.744851 5755 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:20.745568 5769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:20.745735 5755 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:20.745834 5755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "1170a754e75f4fc9b538abe78b483eae"
format_stamp: "Formatted at 2026-09-27 03:28:20 on dist-test-slave-j0hf"
server_key: "3b274a59cc7940dbe37585dd436b4580"
server_key_iv: "e127c152ae37b154f7c26f789dfaf66b"
server_key_version: "encryptionkey@0"
I20260927 03:28:20.745914 5755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:20.799721 5755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:20.800061 5755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:20.800222 5755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:20.804242 5755 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34729
I20260927 03:28:20.804284 5821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34729 every 8 connection(s)
I20260927 03:28:20.804654 5755 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:28:20.805052 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 5755
I20260927 03:28:20.805154 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:28:20.805384 392 external_mini_cluster.cc:1428] Setting key 110d6073e6536af1c95faff769416faa
I20260927 03:28:20.805434 5822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:20.808597 5822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1170a754e75f4fc9b538abe78b483eae: Bootstrap starting.
I20260927 03:28:20.809195 5822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1170a754e75f4fc9b538abe78b483eae: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:20.809479 5822 log.cc:824] T 00000000000000000000000000000000 P 1170a754e75f4fc9b538abe78b483eae: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:20.810132 5822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1170a754e75f4fc9b538abe78b483eae: No bootstrap required, opened a new log
I20260927 03:28:20.811084 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:20.806000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59876 (local address 127.0.98.62:34729)
0927 03:28:20.806193 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:20.806200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:20.806228 (+ 28us) server_negotiation.cc:407] Connection header received
0927 03:28:20.806836 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:20.806844 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:20.806946 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:20.807115 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:20.807322 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:20.808255 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:20.809143 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:20.809630 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:20.809770 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:20.810461 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:20.810466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:20.810466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:28:20.810635 (+ 169us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:20.810667 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:28:20.810677 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:20.810753 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:20.810829 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 03:28:20.810916 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":97,"threads_started":1}
I20260927 03:28:20.812135 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:28:20.812384 5822 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1170a754e75f4fc9b538abe78b483eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1170a754e75f4fc9b538abe78b483eae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34729 } }
I20260927 03:28:20.812536 5822 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1170a754e75f4fc9b538abe78b483eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:20.812587 5822 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1170a754e75f4fc9b538abe78b483eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1170a754e75f4fc9b538abe78b483eae, State: Initialized, Role: FOLLOWER
I20260927 03:28:20.812708 5822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1170a754e75f4fc9b538abe78b483eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1170a754e75f4fc9b538abe78b483eae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34729 } }
I20260927 03:28:20.812798 5822 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1170a754e75f4fc9b538abe78b483eae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:28:20.812835 5822 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1170a754e75f4fc9b538abe78b483eae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:28:20.812889 5822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1170a754e75f4fc9b538abe78b483eae [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:20.813514 5822 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1170a754e75f4fc9b538abe78b483eae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1170a754e75f4fc9b538abe78b483eae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34729 } }
I20260927 03:28:20.813647 5822 leader_election.cc:302] T 00000000000000000000000000000000 P 1170a754e75f4fc9b538abe78b483eae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1170a754e75f4fc9b538abe78b483eae; no voters:
I20260927 03:28:20.813839 5822 leader_election.cc:288] T 00000000000000000000000000000000 P 1170a754e75f4fc9b538abe78b483eae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:28:20.813875 5827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1170a754e75f4fc9b538abe78b483eae [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:20.814054 5827 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1170a754e75f4fc9b538abe78b483eae [term 1 LEADER]: Becoming Leader. State: Replica: 1170a754e75f4fc9b538abe78b483eae, State: Running, Role: LEADER
I20260927 03:28:20.814096 5822 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1170a754e75f4fc9b538abe78b483eae [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:28:20.814182 5827 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1170a754e75f4fc9b538abe78b483eae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1170a754e75f4fc9b538abe78b483eae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34729 } }
I20260927 03:28:20.814554 5829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1170a754e75f4fc9b538abe78b483eae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1170a754e75f4fc9b538abe78b483eae. Latest consensus state: current_term: 1 leader_uuid: "1170a754e75f4fc9b538abe78b483eae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1170a754e75f4fc9b538abe78b483eae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34729 } } }
I20260927 03:28:20.814566 5828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1170a754e75f4fc9b538abe78b483eae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1170a754e75f4fc9b538abe78b483eae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1170a754e75f4fc9b538abe78b483eae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34729 } } }
I20260927 03:28:20.814663 5828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1170a754e75f4fc9b538abe78b483eae [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:20.814659 5829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1170a754e75f4fc9b538abe78b483eae [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:20.814922 5835 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:28:20.815402 5835 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:28:20.816830 5835 catalog_manager.cc:1347] Generated new cluster ID: 0b0ce8c470ab4a22b8a3af8d2f4322da
I20260927 03:28:20.816886 5835 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:28:20.846275 5835 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:28:20.847064 5835 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:28:20.853459 5835 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1170a754e75f4fc9b538abe78b483eae: Generated new TSK 0
I20260927 03:28:20.853686 5835 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:28:20.872117 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:20.868308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59890 (local address 127.0.98.62:34729)
0927 03:28:20.868375 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:20.868378 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:20.868473 (+ 95us) server_negotiation.cc:407] Connection header received
0927 03:28:20.868535 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:20.868536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:20.868551 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:20.868636 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0927 03:28:20.868759 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:20.869516 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:20.870290 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:20.870595 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:20.870667 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:20.870849 (+ 182us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 03:28:20.871897 (+ 1048us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36493/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 03:28:20.872039 (+ 142us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59890: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36493/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 03:28:20.872189 5825 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59890: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36493/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 03:28:20.874106 5851 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:20.868220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34729 (local address 127.0.0.1:59890)
0927 03:28:20.868404 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:20.868414 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:20.868470 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:20.868640 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:20.868640 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:20.868642 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 03:28:20.868699 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:20.868700 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:20.869597 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:20.869598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:20.870202 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:20.870204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:20.870680 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:20.870688 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:20.870734 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:20.870739 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 03:28:20.872006 (+ 1267us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36493/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 03:28:20.873995 (+ 1989us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34729: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36493/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":1768,"thread_start_us":108,"threads_started":1}
I20260927 03:28:20.874534 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 5755
E20260927 03:28:20.884876 5852 webserver.cc:533] Webserver: sslize failed:
E20260927 03:28:20.884980 5852 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T03:28:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (271 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T03:28:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:28:20Z Disabled control of system clock
I20260927 03:28:20.922335 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37455
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37455
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479675865103-392-0/testchainca.cert with env {}
W20260927 03:28:21.005335 5856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:21.005575 5856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:21.005599 5856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:21.006748 5856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:28:21.006789 5856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:21.006804 5856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:21.006819 5856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:28:21.006906 5856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 03:28:21.006920 5856 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 03:28:21.006933 5856 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 03:28:21.008455 5856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37455
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479675865103-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:21.008671 5856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:21.008857 5856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:21.013729 5861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:21.013911 5856 server_base.cc:1034] running on GCE node
W20260927 03:28:21.013724 5864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:21.013729 5862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:21.014281 5856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:21.014528 5856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:21.015664 5856 hybrid_clock.cc:648] HybridClock initialized: now 1790479701015646 us; error 40 us; skew 500 ppm
I20260927 03:28:21.016927 5856 webserver.cc:460] Webserver started at http://127.0.98.62:35773/ using document root <none> and password file <none>
I20260927 03:28:21.017167 5856 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:21.017215 5856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:21.017380 5856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:21.018442 5856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "79973b9c01ef4b6993ed58895460cfcc"
format_stamp: "Formatted at 2026-09-27 03:28:21 on dist-test-slave-j0hf"
server_key: "6401b3d528a56c07a0306b5641d571f8"
server_key_iv: "b6e3d1978ca8b3511646005373a7701d"
server_key_version: "encryptionkey@0"
I20260927 03:28:21.018814 5856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "79973b9c01ef4b6993ed58895460cfcc"
format_stamp: "Formatted at 2026-09-27 03:28:21 on dist-test-slave-j0hf"
server_key: "6401b3d528a56c07a0306b5641d571f8"
server_key_iv: "b6e3d1978ca8b3511646005373a7701d"
server_key_version: "encryptionkey@0"
I20260927 03:28:21.020159 5856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:28:21.020846 5870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:21.021003 5856 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:28:21.021109 5856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "79973b9c01ef4b6993ed58895460cfcc"
format_stamp: "Formatted at 2026-09-27 03:28:21 on dist-test-slave-j0hf"
server_key: "6401b3d528a56c07a0306b5641d571f8"
server_key_iv: "b6e3d1978ca8b3511646005373a7701d"
server_key_version: "encryptionkey@0"
I20260927 03:28:21.021180 5856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:21.043308 5856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:21.043689 5856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:21.043859 5856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:21.047894 5856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37455
I20260927 03:28:21.047940 5922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37455 every 8 connection(s)
I20260927 03:28:21.048323 5856 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:28:21.049257 5923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:21.051642 5923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79973b9c01ef4b6993ed58895460cfcc: Bootstrap starting.
I20260927 03:28:21.052196 5923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79973b9c01ef4b6993ed58895460cfcc: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:21.052428 5923 log.cc:824] T 00000000000000000000000000000000 P 79973b9c01ef4b6993ed58895460cfcc: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:21.053067 5923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79973b9c01ef4b6993ed58895460cfcc: No bootstrap required, opened a new log
I20260927 03:28:21.054620 5923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79973b9c01ef4b6993ed58895460cfcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79973b9c01ef4b6993ed58895460cfcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37455 } }
I20260927 03:28:21.054755 5923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79973b9c01ef4b6993ed58895460cfcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:21.054777 5923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79973b9c01ef4b6993ed58895460cfcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79973b9c01ef4b6993ed58895460cfcc, State: Initialized, Role: FOLLOWER
I20260927 03:28:21.054915 5923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79973b9c01ef4b6993ed58895460cfcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79973b9c01ef4b6993ed58895460cfcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37455 } }
I20260927 03:28:21.055003 5923 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79973b9c01ef4b6993ed58895460cfcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:28:21.055027 5923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79973b9c01ef4b6993ed58895460cfcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:28:21.055055 5923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79973b9c01ef4b6993ed58895460cfcc [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:21.055578 5923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79973b9c01ef4b6993ed58895460cfcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79973b9c01ef4b6993ed58895460cfcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37455 } }
I20260927 03:28:21.055660 5923 leader_election.cc:302] T 00000000000000000000000000000000 P 79973b9c01ef4b6993ed58895460cfcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79973b9c01ef4b6993ed58895460cfcc; no voters:
I20260927 03:28:21.055783 5923 leader_election.cc:288] T 00000000000000000000000000000000 P 79973b9c01ef4b6993ed58895460cfcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:28:21.055855 5926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79973b9c01ef4b6993ed58895460cfcc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:21.056013 5926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79973b9c01ef4b6993ed58895460cfcc [term 1 LEADER]: Becoming Leader. State: Replica: 79973b9c01ef4b6993ed58895460cfcc, State: Running, Role: LEADER
I20260927 03:28:21.056059 5923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79973b9c01ef4b6993ed58895460cfcc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:28:21.056115 5926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79973b9c01ef4b6993ed58895460cfcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79973b9c01ef4b6993ed58895460cfcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37455 } }
I20260927 03:28:21.056507 5927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79973b9c01ef4b6993ed58895460cfcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79973b9c01ef4b6993ed58895460cfcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79973b9c01ef4b6993ed58895460cfcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37455 } } }
I20260927 03:28:21.056938 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 5856
I20260927 03:28:21.057085 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:28:21.057293 392 external_mini_cluster.cc:1428] Setting key 4e2b99ff028f462d8a1a417c6bff5bd2
I20260927 03:28:21.056649 5928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79973b9c01ef4b6993ed58895460cfcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79973b9c01ef4b6993ed58895460cfcc. Latest consensus state: current_term: 1 leader_uuid: "79973b9c01ef4b6993ed58895460cfcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79973b9c01ef4b6993ed58895460cfcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37455 } } }
I20260927 03:28:21.056658 5927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79973b9c01ef4b6993ed58895460cfcc [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:21.057608 5928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79973b9c01ef4b6993ed58895460cfcc [sys.catalog]: This master's current role is: LEADER
W20260927 03:28:21.057924 5941 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79973b9c01ef4b6993ed58895460cfcc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:28:21.057986 5941 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:28:21.058040 5942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:28:21.058218 5942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:28:21.059774 5942 catalog_manager.cc:1347] Generated new cluster ID: 0eac030711ca4974bee5c88e835ed23a
I20260927 03:28:21.059823 5942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:28:21.063271 5945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:21.057822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54858 (local address 127.0.98.62:37455)
0927 03:28:21.058056 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:21.058066 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:21.058097 (+ 31us) server_negotiation.cc:407] Connection header received
0927 03:28:21.058218 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:21.058226 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:21.058308 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:21.058449 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:21.058692 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:21.059957 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:21.061053 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:21.061644 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:21.061753 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:21.062470 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:21.062473 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:21.062474 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:28:21.062697 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:21.062735 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:28:21.062748 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:21.062808 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:21.062908 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 03:28:21.063072 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":65,"threads_started":1}
I20260927 03:28:21.064183 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:28:21.069746 5942 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:28:21.070353 5942 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:28:21.073529 5942 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79973b9c01ef4b6993ed58895460cfcc: Generated new TSK 0
I20260927 03:28:21.073735 5942 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:28:21.532236 5945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:21.082294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54872 (local address 127.0.98.62:37455)
0927 03:28:21.082368 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:21.082370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:21.082402 (+ 32us) server_negotiation.cc:407] Connection header received
0927 03:28:21.082471 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:21.082472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:21.082487 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:21.082587 (+ 100us) server_negotiation.cc:226] Negotiated authn=JWT
0927 03:28:21.082705 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:21.083642 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:21.084456 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:21.084844 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:21.084919 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:21.085093 (+ 174us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 03:28:21.531985 (+446892us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 03:28:21.532172 (+ 187us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54872: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 03:28:21.532305 5945 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54872: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 03:28:21.532363 5952 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:21.082162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37455 (local address 127.0.0.1:54872)
0927 03:28:21.082342 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:21.082358 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:21.082410 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:21.082583 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:21.082584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:21.082586 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 03:28:21.082635 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:21.082637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:21.083749 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:21.083749 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:21.084363 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:21.084365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:21.084937 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:21.084941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:21.084982 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:21.084986 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 03:28:21.532137 (+447151us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 03:28:21.532250 (+ 113us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37455: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":95,"threads_started":1}
I20260927 03:28:21.532704 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 5856
2026-09-27T03:28:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (637 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (89 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (88 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:28:21 dist-test-slave-j0hf krb5kdc[5964](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:28:21 dist-test-slave-j0hf krb5kdc[5964](info): set up 2 sockets
Sep 27 03:28:21 dist-test-slave-j0hf krb5kdc[5964](info): commencing operation
krb5kdc: starting...
W20260927 03:28:22.751787 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:28:22 dist-test-slave-j0hf krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:28:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:28:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:28:22.869831 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40281
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:22.950680 5977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:22.950948 5977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:22.950990 5977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:22.952266 5977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:28:22.952337 5977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:22.952363 5977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:22.952386 5977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:28:22.952417 5977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:28:22.954000 5977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:22.954280 5977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:22.954510 5977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:22.959517 5985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:22.959517 5982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:22.959548 5983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:22.959764 5977 server_base.cc:1034] running on GCE node
I20260927 03:28:22.960005 5977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:22.960307 5977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:22.961468 5977 hybrid_clock.cc:648] HybridClock initialized: now 1790479702961442 us; error 29 us; skew 500 ppm
Sep 27 03:28:22 dist-test-slave-j0hf krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:22.969705 5977 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:28:22.970208 5977 webserver.cc:460] Webserver started at http://127.0.98.62:38969/ using document root <none> and password file <none>
I20260927 03:28:22.970427 5977 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:22.970469 5977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:22.970561 5977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:22.971665 5977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "0290fdd8cff041be9ae9bb0d3e868231"
format_stamp: "Formatted at 2026-09-27 03:28:22 on dist-test-slave-j0hf"
server_key: "cb254d0dc480a6c6b70efb5d4b2efb3c"
server_key_iv: "e975290aa6880bd8d88e506988d8cf63"
server_key_version: "encryptionkey@0"
I20260927 03:28:22.971968 5977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "0290fdd8cff041be9ae9bb0d3e868231"
format_stamp: "Formatted at 2026-09-27 03:28:22 on dist-test-slave-j0hf"
server_key: "cb254d0dc480a6c6b70efb5d4b2efb3c"
server_key_iv: "e975290aa6880bd8d88e506988d8cf63"
server_key_version: "encryptionkey@0"
I20260927 03:28:22.973227 5977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:22.974009 5992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:22.974206 5977 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:22.974316 5977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "0290fdd8cff041be9ae9bb0d3e868231"
format_stamp: "Formatted at 2026-09-27 03:28:22 on dist-test-slave-j0hf"
server_key: "cb254d0dc480a6c6b70efb5d4b2efb3c"
server_key_iv: "e975290aa6880bd8d88e506988d8cf63"
server_key_version: "encryptionkey@0"
I20260927 03:28:22.974406 5977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:23.003932 5977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:23.005963 5977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:23.006130 5977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:23.010033 5977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40281
I20260927 03:28:23.010079 6044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40281 every 8 connection(s)
I20260927 03:28:23.010617 5977 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:28:23.011274 6045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:23.013665 6045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0290fdd8cff041be9ae9bb0d3e868231: Bootstrap starting.
I20260927 03:28:23.014271 6045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0290fdd8cff041be9ae9bb0d3e868231: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:23.014569 6045 log.cc:824] T 00000000000000000000000000000000 P 0290fdd8cff041be9ae9bb0d3e868231: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:23.015056 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 5977
I20260927 03:28:23.015170 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:28:23.015373 392 external_mini_cluster.cc:1428] Setting key e10f6727eeaa8cec9d24d1776104d116
I20260927 03:28:23.015379 6045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0290fdd8cff041be9ae9bb0d3e868231: No bootstrap required, opened a new log
I20260927 03:28:23.018249 6045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0290fdd8cff041be9ae9bb0d3e868231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0290fdd8cff041be9ae9bb0d3e868231" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40281 } }
I20260927 03:28:23.018383 6045 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0290fdd8cff041be9ae9bb0d3e868231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:23.018405 6045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0290fdd8cff041be9ae9bb0d3e868231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0290fdd8cff041be9ae9bb0d3e868231, State: Initialized, Role: FOLLOWER
I20260927 03:28:23.018558 6045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0290fdd8cff041be9ae9bb0d3e868231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0290fdd8cff041be9ae9bb0d3e868231" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40281 } }
I20260927 03:28:23.018625 6045 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0290fdd8cff041be9ae9bb0d3e868231 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:28:23.018646 6045 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0290fdd8cff041be9ae9bb0d3e868231 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:28:23.018713 6045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0290fdd8cff041be9ae9bb0d3e868231 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:23.019404 6045 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0290fdd8cff041be9ae9bb0d3e868231 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0290fdd8cff041be9ae9bb0d3e868231" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40281 } }
I20260927 03:28:23.019543 6045 leader_election.cc:302] T 00000000000000000000000000000000 P 0290fdd8cff041be9ae9bb0d3e868231 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0290fdd8cff041be9ae9bb0d3e868231; no voters:
I20260927 03:28:23.019694 6045 leader_election.cc:288] T 00000000000000000000000000000000 P 0290fdd8cff041be9ae9bb0d3e868231 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:28:23.019740 6050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0290fdd8cff041be9ae9bb0d3e868231 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:23.019905 6050 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0290fdd8cff041be9ae9bb0d3e868231 [term 1 LEADER]: Becoming Leader. State: Replica: 0290fdd8cff041be9ae9bb0d3e868231, State: Running, Role: LEADER
I20260927 03:28:23.019968 6045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0290fdd8cff041be9ae9bb0d3e868231 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:28:23.020030 6050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0290fdd8cff041be9ae9bb0d3e868231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0290fdd8cff041be9ae9bb0d3e868231" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40281 } }
I20260927 03:28:23.020398 6051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0290fdd8cff041be9ae9bb0d3e868231 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0290fdd8cff041be9ae9bb0d3e868231" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0290fdd8cff041be9ae9bb0d3e868231" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40281 } } }
I20260927 03:28:23.020495 6051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0290fdd8cff041be9ae9bb0d3e868231 [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:23.020392 6050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0290fdd8cff041be9ae9bb0d3e868231 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0290fdd8cff041be9ae9bb0d3e868231. Latest consensus state: current_term: 1 leader_uuid: "0290fdd8cff041be9ae9bb0d3e868231" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0290fdd8cff041be9ae9bb0d3e868231" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40281 } } }
I20260927 03:28:23.020618 6050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0290fdd8cff041be9ae9bb0d3e868231 [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:23.021211 6058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:28:23.021703 6058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 03:28:23 dist-test-slave-j0hf krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:23.023522 6058 catalog_manager.cc:1347] Generated new cluster ID: c9ae658490b34b12bc4334abb728ed1c
I20260927 03:28:23.023710 6058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:28:23.027491 6049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:23.015954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45524 (local address 127.0.98.62:40281)
0927 03:28:23.016132 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:23.016140 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:23.016166 (+ 26us) server_negotiation.cc:407] Connection header received
0927 03:28:23.016786 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:23.016794 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:23.016845 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:23.016963 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:23.017723 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:23.018737 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:23.019846 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:23.020514 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:23.020616 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:23.023803 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:23.023807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:23.023808 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:23.023820 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:23.025861 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:23.026248 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:23.026251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:23.026252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:23.026361 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:23.026561 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:23.026562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:23.026562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:23.026875 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:23.027042 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:23.027114 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 03:28:23.027253 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
I20260927 03:28:23.033816 6058 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:28:23.034332 6058 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:28:23.043251 6058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0290fdd8cff041be9ae9bb0d3e868231: Generated new TSK 0
I20260927 03:28:23.043462 6058 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:28:23.068758 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40281
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:23.151615 6071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:23.151819 6071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:23.151844 6071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:23.153038 6071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:23.153083 6071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:23.153136 6071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:28:23.154641 6071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:23.154897 6071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:23.155140 6071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:23.160050 6076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:23.160054 6079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:23.160033 6077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:23.160337 6071 server_base.cc:1034] running on GCE node
I20260927 03:28:23.160581 6071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:23.160818 6071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:23.162082 6071 hybrid_clock.cc:648] HybridClock initialized: now 1790479703162069 us; error 38 us; skew 500 ppm
Sep 27 03:28:23 dist-test-slave-j0hf krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:23.165037 6071 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:28:23.165498 6071 webserver.cc:460] Webserver started at http://127.0.98.1:35407/ using document root <none> and password file <none>
I20260927 03:28:23.165763 6071 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:23.165836 6071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:23.165984 6071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:23.167059 6071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-0/data/instance:
uuid: "30e8de160c55469394589bfb1cc447c4"
format_stamp: "Formatted at 2026-09-27 03:28:23 on dist-test-slave-j0hf"
server_key: "b3cc866c57f8d80d0056490ad89437d3"
server_key_iv: "723b76e189d4a5eb93e8a6fb8b065825"
server_key_version: "encryptionkey@0"
I20260927 03:28:23.167493 6071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-0/wal/instance:
uuid: "30e8de160c55469394589bfb1cc447c4"
format_stamp: "Formatted at 2026-09-27 03:28:23 on dist-test-slave-j0hf"
server_key: "b3cc866c57f8d80d0056490ad89437d3"
server_key_iv: "723b76e189d4a5eb93e8a6fb8b065825"
server_key_version: "encryptionkey@0"
I20260927 03:28:23.168897 6071 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:28:23.169719 6086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:23.169931 6071 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:28:23.170035 6071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "30e8de160c55469394589bfb1cc447c4"
format_stamp: "Formatted at 2026-09-27 03:28:23 on dist-test-slave-j0hf"
server_key: "b3cc866c57f8d80d0056490ad89437d3"
server_key_iv: "723b76e189d4a5eb93e8a6fb8b065825"
server_key_version: "encryptionkey@0"
I20260927 03:28:23.170126 6071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:23.178718 6071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:23.180878 6071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:23.181047 6071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:23.181277 6071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:23.181597 6071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:23.181653 6071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:23.181696 6071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:23.181725 6071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:23.187796 6071 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36195
I20260927 03:28:23.187836 6199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36195 every 8 connection(s)
I20260927 03:28:23.188179 6071 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:28:23.194190 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 6071
I20260927 03:28:23.194304 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:28:23.194535 392 external_mini_cluster.cc:1428] Setting key 99e6ac467dd2f2272a7c6320f2be1df9
Sep 27 03:28:23 dist-test-slave-j0hf krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:23.198865 6202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:23.188911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40281 (local address 127.0.98.1:35597)
0927 03:28:23.189124 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:23.189169 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:23.190123 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:23.190447 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:23.190451 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:23.190727 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:23.190928 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:23.190932 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:23.191812 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:23.191813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:23.192651 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:23.192653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:23.193093 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:23.193094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:23.193201 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:23.193765 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:23.193774 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:23.195599 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:23.197337 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:23.197345 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:23.197348 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:23.197702 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:23.197954 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:23.197955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:23.197955 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:23.198064 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:23.198348 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:23.198352 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:23.198478 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 03:28:23.198572 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 03:28:23.198697 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":93,"threads_started":1}
I20260927 03:28:23.198870 6049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:23.189082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35597 (local address 127.0.98.62:40281)
0927 03:28:23.189133 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:23.189135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:23.189352 (+ 217us) server_negotiation.cc:407] Connection header received
0927 03:28:23.190263 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:23.190264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:23.190276 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:23.190363 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:23.191045 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:23.191714 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:23.192768 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:23.193002 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:23.193078 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:23.195692 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:23.195694 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:23.195694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:23.195703 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:23.197248 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:23.197814 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:23.197815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:23.197816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:23.197887 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:23.198129 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:23.198130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:23.198130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:23.198283 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:23.198353 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:23.198616 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 03:28:23.198714 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 03:28:23.199502 6200 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40281
I20260927 03:28:23.199622 6200 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:23.199813 6200 heartbeater.cc:511] Master 127.0.98.62:40281 requested a full tablet report, sending...
I20260927 03:28:23.200462 6009 ts_manager.cc:194] Registered new tserver with Master: 30e8de160c55469394589bfb1cc447c4 (127.0.98.1:36195)
I20260927 03:28:23.201481 6009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35597
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:28:23.219198 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40281
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:23.300751 6205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:23.300966 6205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:23.300989 6205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:23.302448 6205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:23.302512 6205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:23.302538 6205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:28:23.304090 6205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:23.304291 6205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:23.304476 6205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:23.309721 6213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:23.309712 6211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:23.309712 6210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:23.310019 6205 server_base.cc:1034] running on GCE node
I20260927 03:28:23.310272 6205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:23.310546 6205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:23.311717 6205 hybrid_clock.cc:648] HybridClock initialized: now 1790479703311719 us; error 43 us; skew 500 ppm
Sep 27 03:28:23 dist-test-slave-j0hf krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:23.314356 6205 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:28:23.314857 6205 webserver.cc:460] Webserver started at http://127.0.98.2:43147/ using document root <none> and password file <none>
I20260927 03:28:23.315092 6205 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:23.315160 6205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:23.315310 6205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:23.316491 6205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-1/data/instance:
uuid: "60b2a57bcf5e44e98822bad1ffcdd0d0"
format_stamp: "Formatted at 2026-09-27 03:28:23 on dist-test-slave-j0hf"
server_key: "65a698448cf5c804f6d9ec2a366478cd"
server_key_iv: "7cf0c44ee1676ccf87abad06dd8d6f47"
server_key_version: "encryptionkey@0"
I20260927 03:28:23.316869 6205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-1/wal/instance:
uuid: "60b2a57bcf5e44e98822bad1ffcdd0d0"
format_stamp: "Formatted at 2026-09-27 03:28:23 on dist-test-slave-j0hf"
server_key: "65a698448cf5c804f6d9ec2a366478cd"
server_key_iv: "7cf0c44ee1676ccf87abad06dd8d6f47"
server_key_version: "encryptionkey@0"
I20260927 03:28:23.318226 6205 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:28:23.318994 6220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:23.319185 6205 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:28:23.319283 6205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "60b2a57bcf5e44e98822bad1ffcdd0d0"
format_stamp: "Formatted at 2026-09-27 03:28:23 on dist-test-slave-j0hf"
server_key: "65a698448cf5c804f6d9ec2a366478cd"
server_key_iv: "7cf0c44ee1676ccf87abad06dd8d6f47"
server_key_version: "encryptionkey@0"
I20260927 03:28:23.319366 6205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:23.346876 6205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:23.348882 6205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:23.349009 6205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:23.349192 6205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:23.349512 6205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:23.349551 6205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:23.349575 6205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:23.349587 6205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:23.355473 6205 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34541
I20260927 03:28:23.355566 6333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34541 every 8 connection(s)
I20260927 03:28:23.355877 6205 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:28:23 dist-test-slave-j0hf krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:23.364344 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 6205
I20260927 03:28:23.364498 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:28:23.364683 392 external_mini_cluster.cc:1428] Setting key 4f8cb26ea6dfe22edcf3c6001c4e52e7
I20260927 03:28:23.365861 6049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:23.356623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60825 (local address 127.0.98.62:40281)
0927 03:28:23.356674 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:23.356676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:23.356920 (+ 244us) server_negotiation.cc:407] Connection header received
0927 03:28:23.357854 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:23.357855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:23.357864 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:23.357932 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:23.358502 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:23.358991 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:23.360058 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:23.360262 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:23.360334 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:23.362525 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:23.362527 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:23.362527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:23.362535 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:23.363983 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:23.364572 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:23.364574 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:23.364574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:23.364646 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:23.365024 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:23.365025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:23.365026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:23.365202 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:23.365273 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:23.365612 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 03:28:23.365714 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:28:23.365990 6336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:23.356587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40281 (local address 127.0.98.2:60825)
0927 03:28:23.356764 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:23.356807 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:23.357760 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:23.357996 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:23.358000 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:23.358242 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:23.358430 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:23.358434 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:23.359070 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:23.359071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:23.359952 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:23.359954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:23.360341 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:23.360341 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:23.360438 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:23.360945 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:23.360953 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:23.362428 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:23.364070 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:23.364076 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:23.364078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:23.364453 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:23.364736 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:23.364737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:23.364738 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:23.364927 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:23.365273 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:23.365278 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:23.365484 (+ 206us) client_negotiation.cc:769] Sending connection context
0927 03:28:23.365580 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 03:28:23.365726 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":111,"threads_started":1}
I20260927 03:28:23.366640 6334 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40281
I20260927 03:28:23.366761 6334 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:23.366974 6334 heartbeater.cc:511] Master 127.0.98.62:40281 requested a full tablet report, sending...
I20260927 03:28:23.367466 6009 ts_manager.cc:194] Registered new tserver with Master: 60b2a57bcf5e44e98822bad1ffcdd0d0 (127.0.98.2:34541)
I20260927 03:28:23.368026 6009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60825
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:28:23.391101 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40281
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:23.481706 6339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:23.481964 6339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:23.481993 6339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:23.483224 6339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:23.483269 6339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:23.483300 6339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:28:23.484959 6339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:23.485152 6339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:23.485342 6339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:23.490590 6347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:23.490545 6344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:23.490564 6345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:23.490604 6339 server_base.cc:1034] running on GCE node
I20260927 03:28:23.491104 6339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:23.491354 6339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:23.492520 6339 hybrid_clock.cc:648] HybridClock initialized: now 1790479703492528 us; error 50 us; skew 500 ppm
Sep 27 03:28:23 dist-test-slave-j0hf krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:23.495131 6339 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:28:23.495582 6339 webserver.cc:460] Webserver started at http://127.0.98.3:42403/ using document root <none> and password file <none>
I20260927 03:28:23.495769 6339 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:23.495807 6339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:23.495975 6339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:23.497054 6339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-2/data/instance:
uuid: "90dfaaa1917b4a11bfde4d852a8e3498"
format_stamp: "Formatted at 2026-09-27 03:28:23 on dist-test-slave-j0hf"
server_key: "7ff8d764e4bc938bfd96b3eaf6d98a49"
server_key_iv: "0e92060c2461c2b4e8a72deb948e327d"
server_key_version: "encryptionkey@0"
I20260927 03:28:23.497404 6339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-2/wal/instance:
uuid: "90dfaaa1917b4a11bfde4d852a8e3498"
format_stamp: "Formatted at 2026-09-27 03:28:23 on dist-test-slave-j0hf"
server_key: "7ff8d764e4bc938bfd96b3eaf6d98a49"
server_key_iv: "0e92060c2461c2b4e8a72deb948e327d"
server_key_version: "encryptionkey@0"
I20260927 03:28:23.498715 6339 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:28:23.499528 6354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:23.499680 6339 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:23.499764 6339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "90dfaaa1917b4a11bfde4d852a8e3498"
format_stamp: "Formatted at 2026-09-27 03:28:23 on dist-test-slave-j0hf"
server_key: "7ff8d764e4bc938bfd96b3eaf6d98a49"
server_key_iv: "0e92060c2461c2b4e8a72deb948e327d"
server_key_version: "encryptionkey@0"
I20260927 03:28:23.499825 6339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:23.520702 6339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:23.523111 6339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:23.523330 6339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:23.523613 6339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:23.523947 6339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:23.523980 6339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:23.524034 6339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:23.524061 6339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:23.529872 6339 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36267
I20260927 03:28:23.529940 6467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36267 every 8 connection(s)
I20260927 03:28:23.530296 6339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:28:23 dist-test-slave-j0hf krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:23.536571 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 6339
I20260927 03:28:23.536635 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:28:23.536796 392 external_mini_cluster.cc:1428] Setting key 55d2fd4ece96b9a1d7bc99c0dcf3a063
I20260927 03:28:23.539588 6049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:23.531024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55189 (local address 127.0.98.62:40281)
0927 03:28:23.531105 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:23.531108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:23.531268 (+ 160us) server_negotiation.cc:407] Connection header received
0927 03:28:23.532256 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:23.532257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:23.532268 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:23.532322 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:23.532933 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:23.533436 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:23.534414 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:23.534612 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:23.534675 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:23.536747 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:23.536749 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:23.536749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:23.536756 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:23.538218 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:23.538628 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:23.538630 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:23.538630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:23.538689 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:23.538938 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:23.538939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:23.538940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:23.539052 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:23.539128 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:23.539395 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 03:28:23.539492 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 03:28:23.539794 6470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:23.530985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40281 (local address 127.0.98.3:55189)
0927 03:28:23.531146 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:23.531181 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:23.532163 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:23.532391 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:23.532394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:23.532650 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:23.532851 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:23.532855 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:23.533514 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:23.533515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:23.534306 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:23.534308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:23.534688 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:23.534688 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:23.534782 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:23.535257 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:23.535265 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:23.536639 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:23.538326 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:23.538332 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:23.538334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:23.538552 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:23.538756 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:23.538758 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:23.538758 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:23.538871 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:23.539117 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:23.539122 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:23.539262 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 03:28:23.539375 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 03:28:23.539554 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260927 03:28:23.540336 6468 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40281
I20260927 03:28:23.540427 6468 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:23.540618 6468 heartbeater.cc:511] Master 127.0.98.62:40281 requested a full tablet report, sending...
I20260927 03:28:23.540970 6009 ts_manager.cc:194] Registered new tserver with Master: 90dfaaa1917b4a11bfde4d852a8e3498 (127.0.98.3:36267)
I20260927 03:28:23.541298 6009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55189
I20260927 03:28:23.548370 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/test-admin.keytab.
Sep 27 03:28:23 dist-test-slave-j0hf krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:23.563771 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 03:28:23.565536 6475 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:23.564726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54580 (local address 127.0.98.1:36195)
0927 03:28:23.564933 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:23.564938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:23.564950 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:28:23.564976 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:23.564978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:23.564991 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:23.565065 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:23.565449 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54580: BlockingRecv error: recv got EOF from 127.0.0.1:54580 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":89,"threads_started":1}
W20260927 03:28:23.565665 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 03:28:23 dist-test-slave-j0hf krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:23.566699 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 03:28:23 dist-test-slave-j0hf krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:23.567776 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 03:28:23 dist-test-slave-j0hf krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 03:28:23.575814 6475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:23.568449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54594 (local address 127.0.98.1:36195)
0927 03:28:23.568507 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:23.568509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:23.568522 (+ 13us) server_negotiation.cc:407] Connection header received
0927 03:28:23.568588 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:23.568589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:23.568600 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:23.568651 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:23.569027 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:23.569753 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:23.570464 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:23.570781 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:23.570837 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:23.573009 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:23.573017 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:23.573017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:23.573024 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:23.574702 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:23.575064 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:23.575067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:23.575067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:23.575118 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:23.575300 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:23.575300 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:23.575301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:23.575479 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:23.575598 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:23.575676 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 03:28:23.575737 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 03:28:23.577852 6475 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:23.577218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54610 (local address 127.0.98.1:36195)
0927 03:28:23.577253 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:23.577255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:23.577347 (+ 92us) server_negotiation.cc:407] Connection header received
0927 03:28:23.577397 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:23.577397 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:23.577407 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:23.577450 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:23.577765 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54610: BlockingRecv error: recv got EOF from 127.0.0.1:54610 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 03:28:23.577960 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 03:28:23 dist-test-slave-j0hf krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:23.579030 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 03:28:23 dist-test-slave-j0hf krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:23.579954 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 03:28:23 dist-test-slave-j0hf krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 03:28:23.587272 6475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:23.580559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54612 (local address 127.0.98.1:36195)
0927 03:28:23.580620 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:23.580621 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:23.580663 (+ 42us) server_negotiation.cc:407] Connection header received
0927 03:28:23.580739 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:23.580740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:23.580752 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:23.580822 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:23.581213 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:23.581716 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:23.582463 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:23.582687 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:23.582778 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:23.584966 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:23.584968 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:23.584968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:23.584975 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:23.586188 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:23.586528 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:23.586529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:23.586530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:23.586586 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:23.586780 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:23.586781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:23.586781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:23.586881 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:23.586978 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:23.587103 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 03:28:23.587178 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 03:28:23.588750 6475 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:23.588315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54620 (local address 127.0.98.1:36195)
0927 03:28:23.588350 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:23.588352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:23.588409 (+ 57us) server_negotiation.cc:407] Connection header received
0927 03:28:23.588457 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:23.588458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:23.588468 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:23.588511 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:23.588721 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54620: BlockingRecv error: recv got EOF from 127.0.0.1:54620 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 03:28:23.588850 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790479675865103-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 03:28:23 dist-test-slave-j0hf krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:23.589802 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 03:28:23 dist-test-slave-j0hf krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:23.590610 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 03:28:23 dist-test-slave-j0hf krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 03:28:23.597432 6475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:23.591129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54628 (local address 127.0.98.1:36195)
0927 03:28:23.591165 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:23.591167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:23.591218 (+ 51us) server_negotiation.cc:407] Connection header received
0927 03:28:23.591266 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:23.591267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:23.591274 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:23.591317 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:23.591788 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:23.592233 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:23.592894 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:23.593067 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:23.593119 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:23.595055 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:23.595056 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:23.595056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:23.595063 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:23.596445 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:23.596755 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:23.596756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:23.596756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:23.596801 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:23.596959 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:23.596960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:23.596960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:23.597041 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:23.597105 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:23.597223 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 03:28:23.597274 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:28:23.598214 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 6071
I20260927 03:28:23.603040 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 6205
I20260927 03:28:23.607592 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 6339
I20260927 03:28:23.611832 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 5977
2026-09-27T03:28:23Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1903 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:28:23 dist-test-slave-j0hf krb5kdc[6488](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:28:23 dist-test-slave-j0hf krb5kdc[6488](info): set up 2 sockets
Sep 27 03:28:23 dist-test-slave-j0hf krb5kdc[6488](info): commencing operation
krb5kdc: starting...
W20260927 03:28:24.647774 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:28:24 dist-test-slave-j0hf krb5kdc[6488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:28:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:28:24Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 03:28:24.766606 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41079
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:24.845230 6501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:24.845432 6501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:24.845454 6501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:24.846613 6501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:28:24.846655 6501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:24.846670 6501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:24.846683 6501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:28:24.846742 6501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:28:24.848230 6501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:24.848470 6501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:24.848726 6501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:24.853443 6509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:24.853466 6506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:24.853473 6507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:24.853717 6501 server_base.cc:1034] running on GCE node
I20260927 03:28:24.853906 6501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:24.854144 6501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:24.855284 6501 hybrid_clock.cc:648] HybridClock initialized: now 1790479704855271 us; error 29 us; skew 500 ppm
Sep 27 03:28:24 dist-test-slave-j0hf krb5kdc[6488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479704, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:24.857759 6501 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 03:28:24.858148 6501 webserver.cc:460] Webserver started at http://127.0.98.62:34559/ using document root <none> and password file <none>
I20260927 03:28:24.858349 6501 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:24.858392 6501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:24.858475 6501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:24.859505 6501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "abc21e02fdc0470ebb9a8fce09d9f081"
format_stamp: "Formatted at 2026-09-27 03:28:24 on dist-test-slave-j0hf"
server_key: "14d5ef482b595efeab526ab022ee839e"
server_key_iv: "5efaa9d48f5d21bf5c45221f60b9e4fb"
server_key_version: "encryptionkey@0"
I20260927 03:28:24.859817 6501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "abc21e02fdc0470ebb9a8fce09d9f081"
format_stamp: "Formatted at 2026-09-27 03:28:24 on dist-test-slave-j0hf"
server_key: "14d5ef482b595efeab526ab022ee839e"
server_key_iv: "5efaa9d48f5d21bf5c45221f60b9e4fb"
server_key_version: "encryptionkey@0"
I20260927 03:28:24.861068 6501 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:28:24.861719 6516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:24.861922 6501 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:28:24.862010 6501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "abc21e02fdc0470ebb9a8fce09d9f081"
format_stamp: "Formatted at 2026-09-27 03:28:24 on dist-test-slave-j0hf"
server_key: "14d5ef482b595efeab526ab022ee839e"
server_key_iv: "5efaa9d48f5d21bf5c45221f60b9e4fb"
server_key_version: "encryptionkey@0"
I20260927 03:28:24.862064 6501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:24.882844 6501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:24.884893 6501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:24.885010 6501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:24.888898 6501 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41079
I20260927 03:28:24.888931 6568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41079 every 8 connection(s)
I20260927 03:28:24.889319 6501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:28:24.890333 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:24.891290 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 6501
I20260927 03:28:24.891382 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:28:24.891608 392 external_mini_cluster.cc:1428] Setting key 3effc562017374d48178409a08c4a9b4
I20260927 03:28:24.892969 6569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081: Bootstrap starting.
I20260927 03:28:24.893589 6569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:24.893883 6569 log.cc:824] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:24.894603 6569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081: No bootstrap required, opened a new log
I20260927 03:28:24.896533 6569 raft_consensus.cc:348] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } }
I20260927 03:28:24.896742 6569 raft_consensus.cc:374] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:24.896798 6569 raft_consensus.cc:729] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abc21e02fdc0470ebb9a8fce09d9f081, State: Initialized, Role: FOLLOWER
I20260927 03:28:24.896957 6569 consensus_queue.cc:260] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } }
I20260927 03:28:24.897061 6569 raft_consensus.cc:388] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:28:24.897104 6569 raft_consensus.cc:482] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:28:24.897153 6569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:24.897687 6569 raft_consensus.cc:504] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } }
I20260927 03:28:24.897814 6569 leader_election.cc:302] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abc21e02fdc0470ebb9a8fce09d9f081; no voters:
I20260927 03:28:24.897989 6569 leader_election.cc:288] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:28:24.898059 6574 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:24.898204 6574 raft_consensus.cc:686] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [term 1 LEADER]: Becoming Leader. State: Replica: abc21e02fdc0470ebb9a8fce09d9f081, State: Running, Role: LEADER
I20260927 03:28:24.898258 6569 sys_catalog.cc:564] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:28:24.898320 6574 consensus_queue.cc:237] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } }
Sep 27 03:28:24 dist-test-slave-j0hf krb5kdc[6488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 03:28:24.898628 6574 sys_catalog.cc:455] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abc21e02fdc0470ebb9a8fce09d9f081. Latest consensus state: current_term: 1 leader_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } }
I20260927 03:28:24.898718 6574 sys_catalog.cc:458] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:24.898638 6576 sys_catalog.cc:455] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } }
I20260927 03:28:24.898882 6576 sys_catalog.cc:458] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:24.899559 6581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:28:24.900059 6581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:28:24.901700 6581 catalog_manager.cc:1347] Generated new cluster ID: 0d21507db5a54eaaba302abd4d3f957a
I20260927 03:28:24.901750 6581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:28:24.902683 6572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:24.892164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52812 (local address 127.0.98.62:41079)
0927 03:28:24.892348 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:24.892364 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:24.892390 (+ 26us) server_negotiation.cc:407] Connection header received
0927 03:28:24.892803 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:24.892810 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:24.892855 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:24.893019 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:24.893666 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:24.894842 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:24.895602 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:24.896017 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:24.896144 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:24.899100 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:24.899103 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:24.899104 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:24.899113 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:24.901100 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:24.901522 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:24.901525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:24.901525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:24.901668 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:24.901860 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:24.901861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:24.901861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:24.902103 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:24.902256 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:24.902367 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 03:28:24.902503 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
I20260927 03:28:24.911697 6581 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:28:24.912322 6581 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:28:24.918947 6581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081: Generated new TSK 0
I20260927 03:28:24.919164 6581 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 03:28:24.949769 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41079
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:25.046427 6595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:25.046626 6595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:25.046648 6595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:25.047881 6595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:25.047922 6595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:25.047948 6595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:28:25.049374 6595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:25.049612 6595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:25.049872 6595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:25.054589 6601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:25.054574 6600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:25.054800 6595 server_base.cc:1034] running on GCE node
W20260927 03:28:25.054621 6603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:25.055094 6595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:25.055372 6595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:25.056560 6595 hybrid_clock.cc:648] HybridClock initialized: now 1790479705056556 us; error 50 us; skew 500 ppm
Sep 27 03:28:25 dist-test-slave-j0hf krb5kdc[6488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479705, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:25.059010 6595 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 03:28:25.059537 6595 webserver.cc:460] Webserver started at http://127.0.98.1:44803/ using document root <none> and password file <none>
I20260927 03:28:25.059768 6595 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:25.059835 6595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:25.059983 6595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:25.061026 6595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-0/data/instance:
uuid: "38ea6a029ea046ce8eae0d426bf8e485"
format_stamp: "Formatted at 2026-09-27 03:28:25 on dist-test-slave-j0hf"
server_key: "4a354d3de50d0711e06c5f9629284513"
server_key_iv: "3e32b2da851b33358d8220b77405e0d2"
server_key_version: "encryptionkey@0"
I20260927 03:28:25.061385 6595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-0/wal/instance:
uuid: "38ea6a029ea046ce8eae0d426bf8e485"
format_stamp: "Formatted at 2026-09-27 03:28:25 on dist-test-slave-j0hf"
server_key: "4a354d3de50d0711e06c5f9629284513"
server_key_iv: "3e32b2da851b33358d8220b77405e0d2"
server_key_version: "encryptionkey@0"
I20260927 03:28:25.062593 6595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:28:25.063279 6610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:25.063488 6595 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:25.063591 6595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "38ea6a029ea046ce8eae0d426bf8e485"
format_stamp: "Formatted at 2026-09-27 03:28:25 on dist-test-slave-j0hf"
server_key: "4a354d3de50d0711e06c5f9629284513"
server_key_iv: "3e32b2da851b33358d8220b77405e0d2"
server_key_version: "encryptionkey@0"
I20260927 03:28:25.063678 6595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:25.080133 6595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:25.082077 6595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:25.082253 6595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:25.082509 6595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:25.082834 6595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:25.082899 6595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:25.082942 6595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:25.082971 6595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:25.089033 6595 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39213
I20260927 03:28:25.089073 6723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39213 every 8 connection(s)
I20260927 03:28:25.089444 6595 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:28:25 dist-test-slave-j0hf krb5kdc[6488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479705, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 03:28:25.095669 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 6595
I20260927 03:28:25.095799 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:28:25.096019 392 external_mini_cluster.cc:1428] Setting key 601f6717cf272d3bca4675bc03026f39
I20260927 03:28:25.099623 6572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:25.090200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56929 (local address 127.0.98.62:41079)
0927 03:28:25.090283 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:25.090286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:25.090461 (+ 175us) server_negotiation.cc:407] Connection header received
0927 03:28:25.090929 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:25.090930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:25.090940 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:25.090995 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:25.092057 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:25.092610 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:25.093596 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:25.093801 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:25.093870 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:25.096132 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:25.096134 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:25.096134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:25.096143 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:25.097945 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:25.098437 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:25.098439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:25.098439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:25.098514 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:25.098889 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:25.098890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:25.098890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:25.099012 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:25.099091 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:25.099418 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 03:28:25.099554 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 03:28:25.099900 6726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:25.090154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.98.1:56929)
0927 03:28:25.090338 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:25.090374 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:25.090842 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:25.091055 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:25.091058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:25.091742 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:25.091945 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:25.091950 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:25.092696 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:25.092697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:25.093483 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:25.093485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:25.093887 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:25.093888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:25.093985 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:25.094479 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:25.094487 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:25.096015 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:25.098032 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:25.098039 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:25.098041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:25.098312 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:25.098600 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:25.098601 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:25.098602 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:25.098765 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:25.099080 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:25.099085 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:25.099265 (+ 180us) client_negotiation.cc:769] Sending connection context
0927 03:28:25.099399 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 03:28:25.099612 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":89,"threads_started":1}
I20260927 03:28:25.100432 6724 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41079
I20260927 03:28:25.100531 6724 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:25.100728 6724 heartbeater.cc:511] Master 127.0.98.62:41079 requested a full tablet report, sending...
I20260927 03:28:25.101317 6533 ts_manager.cc:194] Registered new tserver with Master: 38ea6a029ea046ce8eae0d426bf8e485 (127.0.98.1:39213)
I20260927 03:28:25.102334 6533 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56929
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 03:28:25.122216 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41079
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:25.204380 6729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:25.204627 6729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:25.204667 6729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:25.205843 6729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:25.205915 6729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:25.205962 6729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:28:25.207595 6729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:25.207865 6729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:25.208110 6729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:25.213258 6734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:25.213387 6735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:25.213407 6729 server_base.cc:1034] running on GCE node
W20260927 03:28:25.213482 6737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:25.213721 6729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:25.213944 6729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:25.215086 6729 hybrid_clock.cc:648] HybridClock initialized: now 1790479705215073 us; error 30 us; skew 500 ppm
Sep 27 03:28:25 dist-test-slave-j0hf krb5kdc[6488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479705, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:25.217710 6729 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 03:28:25.218220 6729 webserver.cc:460] Webserver started at http://127.0.98.2:43127/ using document root <none> and password file <none>
I20260927 03:28:25.218451 6729 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:25.218497 6729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:25.218667 6729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:25.219851 6729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/data/instance:
uuid: "adc03843909b4012b156274ce3ca2535"
format_stamp: "Formatted at 2026-09-27 03:28:25 on dist-test-slave-j0hf"
server_key: "e396ed988ea0f9b2ade731cc3fa9fb69"
server_key_iv: "f10bbae4e293338d1e493c79b4170b07"
server_key_version: "encryptionkey@0"
I20260927 03:28:25.220237 6729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/wal/instance:
uuid: "adc03843909b4012b156274ce3ca2535"
format_stamp: "Formatted at 2026-09-27 03:28:25 on dist-test-slave-j0hf"
server_key: "e396ed988ea0f9b2ade731cc3fa9fb69"
server_key_iv: "f10bbae4e293338d1e493c79b4170b07"
server_key_version: "encryptionkey@0"
I20260927 03:28:25.221586 6729 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 03:28:25.222327 6744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:25.222549 6729 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:25.222645 6729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "adc03843909b4012b156274ce3ca2535"
format_stamp: "Formatted at 2026-09-27 03:28:25 on dist-test-slave-j0hf"
server_key: "e396ed988ea0f9b2ade731cc3fa9fb69"
server_key_iv: "f10bbae4e293338d1e493c79b4170b07"
server_key_version: "encryptionkey@0"
I20260927 03:28:25.222718 6729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:25.238262 6729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:25.240216 6729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:25.240371 6729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:25.240614 6729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:25.240935 6729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:25.240990 6729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:25.241041 6729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:25.241071 6729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:25.247066 6729 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44295
I20260927 03:28:25.247184 6857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44295 every 8 connection(s)
I20260927 03:28:25.247470 6729 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:28:25 dist-test-slave-j0hf krb5kdc[6488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479705, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 03:28:25.257158 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 6729
I20260927 03:28:25.257280 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:28:25.257515 392 external_mini_cluster.cc:1428] Setting key c9bcc7b2a48ad39887cd1be61583d143
I20260927 03:28:25.259490 6572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:25.250444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40575 (local address 127.0.98.62:41079)
0927 03:28:25.250521 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:25.250524 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:25.250780 (+ 256us) server_negotiation.cc:407] Connection header received
0927 03:28:25.251572 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:25.251573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:25.251586 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:25.251660 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:25.252624 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:25.253139 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:25.254162 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:25.254376 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:25.254455 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:25.256541 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:25.256542 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:25.256542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:25.256550 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:25.258028 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:25.258484 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:25.258485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:25.258486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:25.258553 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:25.258821 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:25.258822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:25.258823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:25.258942 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:25.259021 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:25.259292 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 03:28:25.259379 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 03:28:25.259809 6860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:25.250419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.98.2:40575)
0927 03:28:25.250602 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:25.250644 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:25.251404 (+ 760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:25.251734 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:25.251738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:25.252329 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:25.252522 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:25.252527 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:25.253244 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:25.253245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:25.254067 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:25.254069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:25.254460 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:25.254461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:25.254593 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:25.255048 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:25.255057 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:25.256425 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:25.258163 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:25.258169 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:25.258171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:25.258395 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:25.258620 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:25.258621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:25.258621 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:25.258736 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:25.259012 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:25.259018 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:25.259161 (+ 143us) client_negotiation.cc:769] Sending connection context
0927 03:28:25.259273 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 03:28:25.259462 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
I20260927 03:28:25.260242 6858 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41079
I20260927 03:28:25.260335 6858 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:25.260536 6858 heartbeater.cc:511] Master 127.0.98.62:41079 requested a full tablet report, sending...
I20260927 03:28:25.260901 6533 ts_manager.cc:194] Registered new tserver with Master: adc03843909b4012b156274ce3ca2535 (127.0.98.2:44295)
I20260927 03:28:25.261296 6533 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40575
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260927 03:28:26.103300 6724 heartbeater.cc:503] Master 127.0.98.62:41079 was elected leader, sending a full tablet report...
I20260927 03:28:26.262199 6858 heartbeater.cc:503] Master 127.0.98.62:41079 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/oryx_127.0.98.3.keytab.
W20260927 03:28:26.319324 392 mini_kdc.cc:262] Time spent creating service keytab for oryx/127.0.98.3: real 1.061s user 0.000s sys 0.003s
I20260927 03:28:26.319752 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41079
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:26.401645 6863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:26.401897 6863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:26.401939 6863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:26.403146 6863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:26.403218 6863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:26.403263 6863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:28:26.404878 6863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:26.405164 6863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:26.405421 6863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:26.410662 6868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:26.410722 6869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:26.410857 6863 server_base.cc:1034] running on GCE node
W20260927 03:28:26.410663 6871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:26.411118 6863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:26.411398 6863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:26.412633 6863 hybrid_clock.cc:648] HybridClock initialized: now 1790479706412607 us; error 37 us; skew 500 ppm
Sep 27 03:28:26 dist-test-slave-j0hf krb5kdc[6488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479706, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:26.415095 6863 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 03:28:26.415592 6863 webserver.cc:460] Webserver started at http://127.0.98.3:45149/ using document root <none> and password file <none>
I20260927 03:28:26.415834 6863 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:26.415902 6863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:26.416052 6863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:26.417174 6863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/data/instance:
uuid: "972f0abeb34849df82b4eca3335ec62d"
format_stamp: "Formatted at 2026-09-27 03:28:26 on dist-test-slave-j0hf"
server_key: "0a162aa99db5bc5f7dbe570217d6f1b7"
server_key_iv: "8975cebbd3a2a5456edde2a6e457f6e6"
server_key_version: "encryptionkey@0"
I20260927 03:28:26.417575 6863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/wal/instance:
uuid: "972f0abeb34849df82b4eca3335ec62d"
format_stamp: "Formatted at 2026-09-27 03:28:26 on dist-test-slave-j0hf"
server_key: "0a162aa99db5bc5f7dbe570217d6f1b7"
server_key_iv: "8975cebbd3a2a5456edde2a6e457f6e6"
server_key_version: "encryptionkey@0"
I20260927 03:28:26.418888 6863 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:28:26.419631 6878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:26.419821 6863 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:26.419921 6863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "972f0abeb34849df82b4eca3335ec62d"
format_stamp: "Formatted at 2026-09-27 03:28:26 on dist-test-slave-j0hf"
server_key: "0a162aa99db5bc5f7dbe570217d6f1b7"
server_key_iv: "8975cebbd3a2a5456edde2a6e457f6e6"
server_key_version: "encryptionkey@0"
I20260927 03:28:26.420003 6863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:26.437156 6863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:26.439095 6863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:26.439273 6863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:26.439644 6863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:26.440009 6863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:26.440064 6863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:26.440107 6863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:26.440133 6863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:26.446281 6863 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34339
I20260927 03:28:26.446326 6991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34339 every 8 connection(s)
I20260927 03:28:26.446712 6863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:28:26 dist-test-slave-j0hf krb5kdc[6488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479706, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 03:28:26.454680 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 6863
I20260927 03:28:26.454798 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:28:26.455049 392 external_mini_cluster.cc:1428] Setting key 203c0083b79f967557947d283dfcdb9d
I20260927 03:28:26.456979 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.447544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44267 (local address 127.0.98.62:41079)
0927 03:28:26.447697 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:26.447699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:26.447851 (+ 152us) server_negotiation.cc:407] Connection header received
0927 03:28:26.448382 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:26.448384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:26.448397 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:26.448472 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:26.449434 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.449955 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.450990 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.451196 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.451284 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.453709 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:26.453711 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:26.453711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:26.453719 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:26.455510 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.455966 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.455968 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.455968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.456039 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.456306 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.456307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.456307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.456429 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:26.456497 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:26.456782 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 03:28:26.456874 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":71,"threads_started":1}
I20260927 03:28:26.457156 6995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.447530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.98.3:44267)
0927 03:28:26.447707 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:26.447760 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:26.448270 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:26.448559 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:26.448563 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:26.449134 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:26.449326 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:26.449330 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.450061 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.450063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:26.450882 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:26.450884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.451294 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.451295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:26.451496 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.452107 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:26.452115 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:26.453608 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:26.455605 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:26.455612 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:26.455614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:26.455872 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:26.456108 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:26.456109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:26.456109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:26.456227 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:26.456508 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:26.456514 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:26.456649 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 03:28:26.456761 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 03:28:26.456917 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":77,"threads_started":1}
I20260927 03:28:26.457638 6992 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41079
I20260927 03:28:26.457749 6992 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:26.457945 6992 heartbeater.cc:511] Master 127.0.98.62:41079 requested a full tablet report, sending...
I20260927 03:28:26.458385 6533 ts_manager.cc:194] Registered new tserver with Master: 972f0abeb34849df82b4eca3335ec62d (127.0.98.3:34339)
I20260927 03:28:26.458745 6533 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44267
I20260927 03:28:26.467137 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260927 03:28:26.493721 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34729
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41079,127.0.98.61:34729
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:26.575318 6998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:26.575567 6998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:26.575589 6998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:26.576673 6998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:28:26.576705 6998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:26.576720 6998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:26.576732 6998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:28:26.576745 6998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:28:26.578143 6998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41079,127.0.98.61:34729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
W20260927 03:28:26.578241 6998 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:41079,127.0.98.61:34729'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 03:28:26.578351 6998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:26.578524 6998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:26.583261 7003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:26.583261 7006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:26.583516 6998 server_base.cc:1034] running on GCE node
W20260927 03:28:26.583385 7004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:26.583865 6998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:26.584127 6998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:26.585268 6998 hybrid_clock.cc:648] HybridClock initialized: now 1790479706585238 us; error 52 us; skew 500 ppm
Sep 27 03:28:26 dist-test-slave-j0hf krb5kdc[6488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479706, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:26.587913 6998 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 03:28:26.588425 6998 webserver.cc:460] Webserver started at http://127.0.98.61:45071/ using document root <none> and password file <none>
I20260927 03:28:26.588662 6998 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:26.588732 6998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:26.588892 6998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:26.589958 6998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-1/data/instance:
uuid: "ab1b1713c6064c04a39dd6f981745533"
format_stamp: "Formatted at 2026-09-27 03:28:26 on dist-test-slave-j0hf"
server_key: "c47d9d80be8b55d7c8ea35dd6bbec74f"
server_key_iv: "b64dac707a7d9252ed42809d1891b916"
server_key_version: "encryptionkey@0"
I20260927 03:28:26.590351 6998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-1/wal/instance:
uuid: "ab1b1713c6064c04a39dd6f981745533"
format_stamp: "Formatted at 2026-09-27 03:28:26 on dist-test-slave-j0hf"
server_key: "c47d9d80be8b55d7c8ea35dd6bbec74f"
server_key_iv: "b64dac707a7d9252ed42809d1891b916"
server_key_version: "encryptionkey@0"
I20260927 03:28:26.591753 6998 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:28:26.592519 7013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:26.592692 6998 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:26.592794 6998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-1/wal
uuid: "ab1b1713c6064c04a39dd6f981745533"
format_stamp: "Formatted at 2026-09-27 03:28:26 on dist-test-slave-j0hf"
server_key: "c47d9d80be8b55d7c8ea35dd6bbec74f"
server_key_iv: "b64dac707a7d9252ed42809d1891b916"
server_key_version: "encryptionkey@0"
I20260927 03:28:26.592873 6998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:26.609851 6998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:26.612402 6998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:26.612569 6998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:26.616624 6998 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34729
I20260927 03:28:26.616675 7065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34729 every 8 connection(s)
I20260927 03:28:26.617058 6998 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-1/data/info.pb
I20260927 03:28:26.618089 7066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:26.618700 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 6998
I20260927 03:28:26.620321 7066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:28:26 dist-test-slave-j0hf krb5kdc[6488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479706, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 03:28:26 dist-test-slave-j0hf krb5kdc[6488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 03:28:26.629534 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.621105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45574 (local address 127.0.98.62:41079)
0927 03:28:26.621177 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:26.621179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:26.621348 (+ 169us) server_negotiation.cc:407] Connection header received
0927 03:28:26.621415 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:26.621416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:26.621424 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:26.621471 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:26.622026 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.622541 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.623410 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.623714 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.623797 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.626289 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:26.626291 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:26.626291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:26.626300 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:26.627999 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.628494 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.628495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.628495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.628560 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.628814 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.628815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.628815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.628979 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:26.629059 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:26.629333 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 03:28:26.629430 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 03:28:26.629733 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.621064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.0.1:45574)
0927 03:28:26.621232 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:26.621279 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:26.621350 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:26.621489 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:26.621492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:26.621822 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:26.621940 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:26.621945 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.622632 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.622633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:26.623313 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:26.623315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.623814 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.623815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:26.623888 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.624535 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:26.624542 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:26.626200 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:26.628102 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:26.628109 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:26.628111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:26.628408 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:26.628628 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:26.628629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:26.628629 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:26.628745 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:26.629049 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:26.629055 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:26.629201 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 03:28:26.629315 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 03:28:26.629470 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":109,"threads_started":1}
I20260927 03:28:26.629755 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.619361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48362 (local address 127.0.98.61:34729)
0927 03:28:26.619572 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:26.619581 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:26.619607 (+ 26us) server_negotiation.cc:407] Connection header received
0927 03:28:26.620020 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:26.620026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:26.620073 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:26.620193 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:26.621042 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.622308 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.623149 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.623551 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.623726 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.626775 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:26.626784 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:26.626784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:26.626791 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:26.628416 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.628833 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.628836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.628837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.628900 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.629122 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.629124 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.629124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.629276 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:26.629446 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:26.629501 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 03:28:26.629586 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":104,"threads_started":1}
I20260927 03:28:26.630535 7066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:28:26 dist-test-slave-j0hf krb5kdc[6488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479706, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 03:28:26.637213 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.630793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34729 (local address 127.0.0.1:48366)
0927 03:28:26.630872 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:26.630885 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:26.630928 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:26.631037 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:26.631037 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:26.631146 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:26.631196 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:26.631197 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.631864 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.631865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:26.632338 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:26.632340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.632662 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.632663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:26.632711 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.633073 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:26.633077 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:26.634252 (+ 1175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:26.636012 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:26.636013 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:26.636013 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:26.636186 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:26.636430 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:26.636431 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:26.636432 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:26.636491 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:26.636773 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:26.636773 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:26.636812 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 03:28:26.636914 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 03:28:26.637069 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":5}
I20260927 03:28:26.637220 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.630836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48366 (local address 127.0.98.61:34729)
0927 03:28:26.630878 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:26.630880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:26.630920 (+ 40us) server_negotiation.cc:407] Connection header received
0927 03:28:26.630965 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:26.630971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:26.630980 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:26.631033 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:26.631253 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.631795 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.632410 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.632598 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.632654 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.634359 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:26.634360 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:26.634361 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:26.634367 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:26.635905 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.636284 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.636285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.636286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.636345 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.636566 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.636567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.636567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.636683 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:26.636781 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:26.636937 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 03:28:26.637122 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":111,"server-negotiator.queue_time_us":23}
I20260927 03:28:26.638772 7066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533: Bootstrap starting.
I20260927 03:28:26.639690 7066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:26.639979 7066 log.cc:824] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:26.640728 7066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533: No bootstrap required, opened a new log
I20260927 03:28:26.642168 7066 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } }
I20260927 03:28:26.642310 7066 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:26.642334 7066 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1b1713c6064c04a39dd6f981745533, State: Initialized, Role: FOLLOWER
I20260927 03:28:26.642467 7066 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } }
I20260927 03:28:26.642824 7066 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:28:26.642791 7072 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } } }
I20260927 03:28:26.642903 7072 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 03:28:26.644919 7083 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:28:26.645030 7083 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:28:26.646311 6998 master_runner.cc:428] Detected that this master ab1b1713c6064c04a39dd6f981745533 is joining an existing cluster
I20260927 03:28:26.646350 6998 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:34729
I20260927 03:28:26.655772 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.647931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48378 (local address 127.0.98.61:34729)
0927 03:28:26.647971 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:26.647973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:26.647985 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:28:26.648069 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:26.648070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:26.648080 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:26.648130 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:26.648743 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.649218 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.650179 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.650461 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.650513 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.651715 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:26.651716 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:26.651717 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:26.651723 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:26.653539 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.654900 (+ 1361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.654901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.654901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.654955 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.655122 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.655123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.655123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.655264 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:26.655332 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:26.655532 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 03:28:26.655659 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":22}
I20260927 03:28:26.655808 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.647392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45586 (local address 127.0.98.62:41079)
0927 03:28:26.647454 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:26.647456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:26.647985 (+ 529us) server_negotiation.cc:407] Connection header received
0927 03:28:26.648140 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:26.648141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:26.648153 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:26.648210 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:26.648738 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.649501 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.650522 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.650833 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.650920 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.652332 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:26.652334 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:26.652334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:26.652343 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:26.654371 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.654893 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.654894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.654895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.654961 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.655185 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.655186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.655187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.655316 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:26.655356 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:26.655614 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 03:28:26.655705 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:28:26.655880 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.647354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.0.1:45586)
0927 03:28:26.647904 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:26.647916 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:26.648011 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:26.648224 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:26.648225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:26.648525 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:26.648649 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:26.648651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.649588 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.649589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:26.650438 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:26.650439 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.650929 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.650930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:26.650999 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.651888 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:26.651893 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:26.652238 (+ 345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:26.654519 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:26.654520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:26.654521 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:26.654778 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:26.655043 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:26.655044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:26.655044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:26.655103 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:26.655400 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:26.655400 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:26.655489 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 03:28:26.655583 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 03:28:26.655719 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":57,"thread_start_us":110,"threads_started":1}
I20260927 03:28:26.656288 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.647758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34729 (local address 127.0.0.1:48378)
0927 03:28:26.647934 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:26.647944 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:26.648001 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:26.648172 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:26.648172 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:26.648521 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:26.648649 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:26.648651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.649302 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.649303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:26.650094 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:26.650096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.650593 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.650594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:26.650632 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.651166 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:26.651171 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:26.651612 (+ 441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:26.654486 (+ 2874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:26.654487 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:26.654488 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:26.654806 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:26.655022 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:26.655022 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:26.655023 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:26.655074 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:26.655337 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:26.655338 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:26.655389 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 03:28:26.655490 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 03:28:26.656187 (+ 697us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":2,"thread_start_us":73,"threads_started":1}
Sep 27 03:28:26 dist-test-slave-j0hf krb5kdc[6488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479704, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 03:28:26.666477 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.658767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48394 (local address 127.0.98.61:34729)
0927 03:28:26.658823 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:26.658826 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:26.659018 (+ 192us) server_negotiation.cc:407] Connection header received
0927 03:28:26.659104 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:26.659105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:26.659113 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:26.659159 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:26.659685 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.660153 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.661306 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.661561 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.661615 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.663791 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:26.663793 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:26.663793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:26.663800 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:26.665240 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.665640 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.665641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.665642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.665701 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.665911 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.665912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.665912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.666027 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:26.666101 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:26.666296 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 03:28:26.666377 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 03:28:26.666494 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.658664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34729 (local address 127.0.0.1:48394)
0927 03:28:26.658898 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:26.658947 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:26.659026 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:26.659164 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:26.659167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:26.659545 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:26.659611 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:26.659613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.660229 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.660231 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:26.661225 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:26.661228 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.661636 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.661637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:26.661721 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.662158 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:26.662163 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:26.663700 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:26.665346 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:26.665353 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:26.665355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:26.665563 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:26.665767 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:26.665768 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:26.665768 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:26.665847 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:26.666094 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:26.666101 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:26.666156 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 03:28:26.666271 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 03:28:26.666386 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":123,"threads_started":1}
I20260927 03:28:26.667071 6533 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:34729: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ab1b1713c6064c04a39dd6f981745533"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 34729
}
attrs {
promote: true
}
}
dest_uuid: "abc21e02fdc0470ebb9a8fce09d9f081"
cas_config_opid_index: -1
I20260927 03:28:26.667524 6533 consensus_queue.cc:237] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: true } }
I20260927 03:28:26.668112 7093 raft_consensus.cc:2932] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ab1b1713c6064c04a39dd6f981745533 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: true } } }
I20260927 03:28:26.668418 7040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:26.669178 7093 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:34729
I20260927 03:28:26.669320 7093 sys_catalog.cc:455] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: true } } }
I20260927 03:28:26.669374 7040 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [term 1 FOLLOWER]: Refusing update from remote peer abc21e02fdc0470ebb9a8fce09d9f081: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 03:28:26.669428 7093 sys_catalog.cc:458] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:26.669472 6998 master.cc:537] Master@127.0.98.61:34729 shutting down...
I20260927 03:28:26.670050 7094 consensus_queue.cc:1035] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 03:28:26.671123 6519 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 03:28:26.671654 6519 consensus_peers.cc:487] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 -> Peer ab1b1713c6064c04a39dd6f981745533 (127.0.98.61:34729): Couldn't send request to peer ab1b1713c6064c04a39dd6f981745533. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 03:28:26.672286 6998 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:28:26.672374 6998 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:26.672425 6998 tablet_replica.cc:330] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533: stopping tablet replica
I20260927 03:28:26.685586 6998 master.cc:552] Master@127.0.98.61:34729 shutdown complete.
I20260927 03:28:26.685645 6998 master_runner.cc:305] Clearing existing system tablet
I20260927 03:28:26.685776 6998 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:26.685822 6998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:26.686569 6998 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.002s sys 0.000s
I20260927 03:28:26.687120 7098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:26.687280 6998 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:26.687341 6998 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:26.687366 6998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-1/wal
uuid: "ab1b1713c6064c04a39dd6f981745533"
format_stamp: "Formatted at 2026-09-27 03:28:26 on dist-test-slave-j0hf"
server_key: "c47d9d80be8b55d7c8ea35dd6bbec74f"
server_key_iv: "b64dac707a7d9252ed42809d1891b916"
server_key_version: "encryptionkey@0"
I20260927 03:28:26.688237 6998 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:26.689731 6998 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 03:28:26.689802 6998 log.cc:1196] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 03:28:26.690011 6998 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533: Deleting consensus metadata
I20260927 03:28:26.690244 6998 master_runner.cc:315] Copying system tablet from 127.0.98.62:41079
I20260927 03:28:26.690855 6998 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:41079
I20260927 03:28:26.697721 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.691252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45594 (local address 127.0.98.62:41079)
0927 03:28:26.691323 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:26.691326 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:26.691395 (+ 69us) server_negotiation.cc:407] Connection header received
0927 03:28:26.691524 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:26.691525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:26.691535 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:26.691593 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:26.691947 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.692471 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.693580 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.693780 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.693837 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.694873 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:26.694874 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:26.694875 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:26.694881 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:26.696371 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.696744 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.696745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.696746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.696811 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.697108 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.697109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.697110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.697295 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:26.697366 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:26.697531 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 03:28:26.697598 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 03:28:26.697717 7104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.691191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.0.1:45594)
0927 03:28:26.691329 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:26.691341 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:26.691411 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:26.691599 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:26.691600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:26.691800 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:26.691875 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:26.691877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.692553 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.692554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:26.693459 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:26.693461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.693855 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.693857 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:26.693934 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.694386 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:26.694391 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:26.694786 (+ 395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:26.696450 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:26.696451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:26.696452 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:26.696669 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:26.696883 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:26.696883 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:26.696884 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:26.696949 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:26.697377 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:26.697378 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:26.697426 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 03:28:26.697501 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 03:28:26.697604 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":81,"threads_started":1}
I20260927 03:28:26.698072 6553 tablet_copy_service.cc:140] P abc21e02fdc0470ebb9a8fce09d9f081: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ab1b1713c6064c04a39dd6f981745533 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:45594)
I20260927 03:28:26.698177 6553 tablet_copy_service.cc:161] P abc21e02fdc0470ebb9a8fce09d9f081: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ab1b1713c6064c04a39dd6f981745533 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:45594: session id = ab1b1713c6064c04a39dd6f981745533-00000000000000000000000000000000
I20260927 03:28:26.699075 6553 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 03:28:26.699707 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 03:28:26.700700 6998 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533: tablet copy: Starting download of 0 data blocks...
I20260927 03:28:26.700872 6998 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533: tablet copy: Starting download of 1 WAL segments...
I20260927 03:28:26.701809 6998 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 03:28:26.702677 6553 tablet_copy_service.cc:342] P abc21e02fdc0470ebb9a8fce09d9f081: Request end of tablet copy session ab1b1713c6064c04a39dd6f981745533-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:45594
I20260927 03:28:26.702742 6553 tablet_copy_service.cc:433] P abc21e02fdc0470ebb9a8fce09d9f081: ending tablet copy session ab1b1713c6064c04a39dd6f981745533-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ab1b1713c6064c04a39dd6f981745533
I20260927 03:28:26.704957 6998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:26.706127 7105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:26.706169 7106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:26.706238 6998 server_base.cc:1034] running on GCE node
W20260927 03:28:26.706295 7108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:26.706453 6998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:26.706595 6998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:26.707717 6998 hybrid_clock.cc:648] HybridClock initialized: now 1790479706707711 us; error 40 us; skew 500 ppm
Sep 27 03:28:26 dist-test-slave-j0hf krb5kdc[6488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479706, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:26.709013 6998 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 03:28:26.709287 6998 webserver.cc:460] Webserver started at http://127.0.98.61:38611/ using document root <none> and password file <none>
I20260927 03:28:26.709409 6998 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:26.709444 6998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:26.710049 6998 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:26.710580 7115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:26.710701 6998 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:26.710747 6998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-1/wal
uuid: "ab1b1713c6064c04a39dd6f981745533"
format_stamp: "Formatted at 2026-09-27 03:28:26 on dist-test-slave-j0hf"
server_key: "c47d9d80be8b55d7c8ea35dd6bbec74f"
server_key_iv: "b64dac707a7d9252ed42809d1891b916"
server_key_version: "encryptionkey@0"
I20260927 03:28:26.710804 6998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:26.719048 6998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:26.720741 6998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:26.723811 6998 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34729
I20260927 03:28:26.724067 7167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34729 every 8 connection(s)
I20260927 03:28:26.724246 6998 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-1/data/info.pb
I20260927 03:28:26.724992 7168 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:28:26.725396 7168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533: Bootstrap starting.
I20260927 03:28:26.727767 7168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:28:26.727995 7168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533: Bootstrap complete.
I20260927 03:28:26.728189 7168 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: true } }
I20260927 03:28:26.728274 7168 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab1b1713c6064c04a39dd6f981745533, State: Initialized, Role: LEARNER
I20260927 03:28:26.728315 7168 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: true } }
I20260927 03:28:26.728502 7168 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:28:26.728493 7171 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: true } } }
I20260927 03:28:26.728590 7171 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [sys.catalog]: This master's current role is: LEARNER
I20260927 03:28:26.729826 7182 catalog_manager.cc:1259] Loaded cluster ID: 0d21507db5a54eaaba302abd4d3f957a
I20260927 03:28:26.729882 7182 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533: loading cluster ID for follower catalog manager: success
I20260927 03:28:26.731002 7182 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533: acquiring CA information for follower catalog manager: success
I20260927 03:28:26.731292 7182 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 03:28:26.737914 7183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.731655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48414 (local address 127.0.98.61:34729)
0927 03:28:26.731772 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:26.731774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:26.731782 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:28:26.731804 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:26.731805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:26.731814 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:26.731897 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:26.732426 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.733070 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.733794 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.734020 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.734095 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.735283 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:26.735285 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:26.735285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:26.735292 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:26.736783 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.737148 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.737150 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.737150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.737203 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.737403 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.737404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.737405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.737516 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:26.737628 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:26.737734 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 03:28:26.737812 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":51,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260927 03:28:26.764451 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40375
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41079,127.0.98.61:34729,127.0.98.60:40375
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:26.855166 7186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:26.855441 7186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:26.855479 7186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:26.856653 7186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:28:26.856721 7186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:26.856745 7186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:26.856767 7186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:28:26.856796 7186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:28:26.858294 7186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41079,127.0.98.61:34729,127.0.98.60:40375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:26.858579 7186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:26.858809 7186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:26.863749 7191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:26.863698 7192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:26.864033 7186 server_base.cc:1034] running on GCE node
W20260927 03:28:26.863698 7194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:26.864356 7186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:26.864705 7186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:26.865857 7186 hybrid_clock.cc:648] HybridClock initialized: now 1790479706865778 us; error 97 us; skew 500 ppm
Sep 27 03:28:26 dist-test-slave-j0hf krb5kdc[6488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479706, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:26.868284 7186 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 03:28:26.868721 7186 webserver.cc:460] Webserver started at http://127.0.98.60:40779/ using document root <none> and password file <none>
I20260927 03:28:26.868963 7186 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:26.869032 7186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:26.869174 7186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:26.870229 7186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-2/data/instance:
uuid: "510a4c04062b4014b9466b0a818c5896"
format_stamp: "Formatted at 2026-09-27 03:28:26 on dist-test-slave-j0hf"
server_key: "7f4b334b5161e1a94057105e202bdb8a"
server_key_iv: "faf043e2a7ac7e52db8abe549254b9ce"
server_key_version: "encryptionkey@0"
I20260927 03:28:26.870599 7186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-2/wal/instance:
uuid: "510a4c04062b4014b9466b0a818c5896"
format_stamp: "Formatted at 2026-09-27 03:28:26 on dist-test-slave-j0hf"
server_key: "7f4b334b5161e1a94057105e202bdb8a"
server_key_iv: "faf043e2a7ac7e52db8abe549254b9ce"
server_key_version: "encryptionkey@0"
I20260927 03:28:26.872040 7186 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:28:26.872743 7201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:26.872905 7186 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:26.873008 7186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-2/wal
uuid: "510a4c04062b4014b9466b0a818c5896"
format_stamp: "Formatted at 2026-09-27 03:28:26 on dist-test-slave-j0hf"
server_key: "7f4b334b5161e1a94057105e202bdb8a"
server_key_iv: "faf043e2a7ac7e52db8abe549254b9ce"
server_key_version: "encryptionkey@0"
I20260927 03:28:26.873080 7186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:26.886654 7186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:26.888729 7186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:26.888942 7186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:26.893007 7186 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40375
I20260927 03:28:26.893055 7253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40375 every 8 connection(s)
I20260927 03:28:26.893433 7186 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-2/data/info.pb
I20260927 03:28:26.894392 7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:26.895917 7254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:28:26.900187 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 7186
Sep 27 03:28:26 dist-test-slave-j0hf krb5kdc[6488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479706, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 03:28:26 dist-test-slave-j0hf krb5kdc[6488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 03:28:26.905980 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.896680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45598 (local address 127.0.98.62:41079)
0927 03:28:26.896751 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:26.896754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:26.896908 (+ 154us) server_negotiation.cc:407] Connection header received
0927 03:28:26.897399 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:26.897400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:26.897411 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:26.897473 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:26.898080 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.898853 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.900020 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.900222 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.900306 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.902560 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:26.902562 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:26.902562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:26.902569 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:26.904127 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.904529 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.904530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.904530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.904586 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.905290 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.905291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.905291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.905417 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:26.905480 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:26.905820 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 03:28:26.905896 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 03:28:26.906359 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.896564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.0.1:45598)
0927 03:28:26.896759 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:26.896807 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:26.897287 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:26.897547 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:26.897550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:26.897790 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:26.897979 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:26.897983 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.898948 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.898949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:26.899919 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:26.899921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.900350 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.900351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:26.900507 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.901072 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:26.901080 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:26.902447 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:26.904215 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:26.904221 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:26.904223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:26.904439 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:26.904659 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:26.904660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:26.904661 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:26.905207 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:26.905493 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:26.905499 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:26.905669 (+ 170us) client_negotiation.cc:769] Sending connection context
0927 03:28:26.905791 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 03:28:26.906002 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":92,"threads_started":1}
I20260927 03:28:26.907083 7254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:28:26.909339 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.900740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57354 (local address 127.0.98.60:40375)
0927 03:28:26.900939 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:26.900944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:26.900967 (+ 23us) server_negotiation.cc:407] Connection header received
0927 03:28:26.900988 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:26.900990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:26.901010 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:26.901091 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:26.901690 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.902429 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.903360 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.903880 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.903959 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.906379 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:26.906384 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:26.906384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:26.906390 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:26.908169 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.908548 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.908550 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.908551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.908599 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.908795 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.908796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.908796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.908915 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:26.909095 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:26.909138 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 03:28:26.909211 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":126,"threads_started":1}
Sep 27 03:28:26 dist-test-slave-j0hf krb5kdc[6488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479706, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 03:28:26.914258 7183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.907637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48416 (local address 127.0.98.61:34729)
0927 03:28:26.907686 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:26.907688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:26.907696 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:28:26.907722 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:26.907723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:26.907735 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:26.907809 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:26.908062 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.908807 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.909597 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.909811 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.909874 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.911725 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:26.911727 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:26.911727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:26.911734 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:26.913138 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.913497 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.913498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.913499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.913560 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.913758 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.913758 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.913759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.913875 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:26.913948 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:26.914095 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 03:28:26.914169 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:28:26.914258 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.907562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34729 (local address 127.0.0.1:48416)
0927 03:28:26.907598 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:26.907607 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:26.907641 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:26.907811 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:26.907812 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:26.907944 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:26.907973 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:26.907974 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.908886 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.908887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:26.909504 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:26.909507 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.909950 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.909951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:26.910009 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.910413 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:26.910417 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:26.911600 (+ 1183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:26.913240 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:26.913241 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:26.913241 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:26.913411 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:26.913634 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:26.913635 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:26.913635 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:26.913695 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:26.913941 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:26.913941 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:26.913984 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 03:28:26.914071 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 03:28:26.914165 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260927 03:28:26.914733 7254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40375 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:28:26 dist-test-slave-j0hf krb5kdc[6488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479706, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 03:28:26.920607 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.914966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40375 (local address 127.0.0.1:57356)
0927 03:28:26.915016 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:26.915026 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:26.915067 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:26.915189 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:26.915189 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:26.915312 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:26.915340 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:26.915341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.915987 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.915988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:26.916446 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:26.916447 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.916826 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.916827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:26.916873 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.917218 (+ 345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:26.917222 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:26.918268 (+ 1046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:26.919610 (+ 1342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:26.919611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:26.919611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:26.919765 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:26.919998 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:26.919998 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:26.919999 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:26.920052 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:26.920301 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:26.920302 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:26.920338 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 03:28:26.920407 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 03:28:26.920498 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 03:28:26.920624 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.915014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57356 (local address 127.0.98.60:40375)
0927 03:28:26.915047 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:26.915049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:26.915057 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:28:26.915121 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:26.915122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:26.915132 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:26.915183 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:26.915414 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.915899 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.916538 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.916744 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.916822 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.918352 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:26.918353 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:26.918353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:26.918360 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:26.919526 (+ 1166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.919873 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.919874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.919874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.919927 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.920130 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.920131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.920131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.920238 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:26.920308 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:26.920430 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 03:28:26.920545 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":19}
I20260927 03:28:26.922106 7254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896: Bootstrap starting.
I20260927 03:28:26.922977 7254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:26.923256 7254 log.cc:824] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:26.924036 7254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896: No bootstrap required, opened a new log
I20260927 03:28:26.925465 7254 raft_consensus.cc:348] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40375 } }
I20260927 03:28:26.925593 7254 raft_consensus.cc:374] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:26.925616 7254 raft_consensus.cc:729] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 510a4c04062b4014b9466b0a818c5896, State: Initialized, Role: FOLLOWER
I20260927 03:28:26.925770 7254 consensus_queue.cc:260] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40375 } }
I20260927 03:28:26.926098 7254 sys_catalog.cc:564] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:28:26.926103 7259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40375 } } }
I20260927 03:28:26.926267 7259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 03:28:26.927992 7270 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:28:26.928051 7270 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:28:26.930227 7186 master_runner.cc:428] Detected that this master 510a4c04062b4014b9466b0a818c5896 is joining an existing cluster
I20260927 03:28:26.930276 7186 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:40375
I20260927 03:28:26.938982 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.931242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.0.1:45610)
0927 03:28:26.931807 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:26.931817 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:26.931909 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:26.932084 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:26.932085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:26.932291 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:26.932352 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:26.932353 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.933233 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.933234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:26.934274 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:26.934275 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.934790 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.934791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:26.934861 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.935366 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:26.935370 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:26.935730 (+ 360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:26.937980 (+ 2250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:26.937981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:26.937981 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:26.938171 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:26.938378 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:26.938378 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:26.938379 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:26.938430 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:26.938689 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:26.938690 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:26.938744 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 03:28:26.938794 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 03:28:26.938902 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":104,"thread_start_us":101,"threads_started":1}
I20260927 03:28:26.939038 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.931273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45610 (local address 127.0.98.62:41079)
0927 03:28:26.931319 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:26.931322 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:26.931886 (+ 564us) server_negotiation.cc:407] Connection header received
0927 03:28:26.931980 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:26.931981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:26.931992 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:26.932022 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:26.932431 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.933151 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.934373 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.934672 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.934712 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.935851 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:26.935852 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:26.935853 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:26.935868 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:26.937876 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.938244 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.938245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.938245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.938310 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.938491 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.938492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.938492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.938625 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:26.938702 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:26.938853 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 03:28:26.938947 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 27 03:28:26 dist-test-slave-j0hf krb5kdc[6488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479704, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 03:28:26.959175 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.941948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40375 (local address 127.0.0.1:57370)
0927 03:28:26.941995 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:26.942009 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:26.942064 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:26.942318 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:26.942319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:26.942475 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:26.942511 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:26.942513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.943185 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.943186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:26.943769 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:26.943771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.944179 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.944179 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:26.944245 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.944659 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:26.944665 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:26.946037 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:26.958065 (+ 12028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:26.958066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:26.958067 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:26.958289 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:26.958505 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:26.958505 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:26.958505 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:26.958562 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:26.958843 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:26.958844 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:26.958902 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 03:28:26.958951 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 03:28:26.959064 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260927 03:28:26.959615 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.942021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57370 (local address 127.0.98.60:40375)
0927 03:28:26.942180 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:26.942181 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:26.942189 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:28:26.942228 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:26.942228 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:26.942236 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:26.942314 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:26.942594 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.943103 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.943869 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.944095 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.944178 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.946120 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:26.946121 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:26.946122 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:26.946129 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:26.957945 (+ 11816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.958378 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.958379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.958380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.958433 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.958629 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.958629 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.958630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.958769 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:26.959362 (+ 593us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:26.959404 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 03:28:26.959506 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":113,"threads_started":1}
I20260927 03:28:26.960287 6533 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:40375: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "510a4c04062b4014b9466b0a818c5896"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 40375
}
attrs {
promote: true
}
}
dest_uuid: "abc21e02fdc0470ebb9a8fce09d9f081"
cas_config_opid_index: 5
I20260927 03:28:26.960467 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.931681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40375 (local address 127.0.0.1:57366)
0927 03:28:26.931939 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:26.931949 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:26.932018 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:26.932165 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:26.932166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:26.932407 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:26.932468 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:26.932470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.933344 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.933345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:26.934266 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:26.934268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.934750 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.934750 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:26.934808 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.955985 (+ 21177us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:26.955990 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:26.956438 (+ 448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:26.959135 (+ 2697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:26.959136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:26.959136 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:26.959377 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:26.959614 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:26.959615 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:26.959615 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:26.959705 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:26.960108 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:26.960108 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:26.960163 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 03:28:26.960260 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 03:28:26.960371 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":49,"thread_start_us":67,"threads_started":1}
I20260927 03:28:26.960567 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.931499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57366 (local address 127.0.98.60:40375)
0927 03:28:26.931535 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:26.931537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:26.931997 (+ 460us) server_negotiation.cc:407] Connection header received
0927 03:28:26.932081 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:26.932081 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:26.932089 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:26.932115 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:26.932550 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.933256 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.934358 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.934635 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.934672 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.956552 (+ 21880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:26.956553 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:26.956554 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:26.956560 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:26.959026 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.959489 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.959490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.959490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.959542 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.959807 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.959808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.959809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.959930 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:26.959972 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:26.960291 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 03:28:26.960455 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":20}
I20260927 03:28:26.960564 6533 consensus_queue.cc:237] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: true } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: true } }
I20260927 03:28:26.961026 7094 raft_consensus.cc:2932] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 510a4c04062b4014b9466b0a818c5896 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: true } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: true } } }
I20260927 03:28:26.961826 7094 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:40375
I20260927 03:28:26.961880 7094 sys_catalog.cc:455] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: true } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: true } } }
I20260927 03:28:26.961927 7094 sys_catalog.cc:458] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:26.962312 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 6595
I20260927 03:28:26.962792 7186 master.cc:537] Master@127.0.98.60:40375 shutting down...
I20260927 03:28:26.962945 7228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:26.963994 7228 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896 [term 1 FOLLOWER]: Refusing update from remote peer abc21e02fdc0470ebb9a8fce09d9f081: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 03:28:26.964649 7093 consensus_queue.cc:1035] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 03:28:26.965701 6517 consensus_peers.cc:487] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 -> Peer 510a4c04062b4014b9466b0a818c5896 (127.0.98.60:40375): Couldn't send request to peer 510a4c04062b4014b9466b0a818c5896. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 03:28:26.966128 7183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.931616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48428 (local address 127.0.98.61:34729)
0927 03:28:26.931713 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:26.931714 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:26.931720 (+ 6us) server_negotiation.cc:407] Connection header received
0927 03:28:26.931739 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:26.931739 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:26.931769 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:26.931827 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:26.932354 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.933121 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.934379 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.934673 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.934712 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.957956 (+ 23244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:26.957958 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:26.957959 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:26.957968 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:26.964533 (+ 6565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.965094 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.965095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.965095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.965169 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:26.965511 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:26.965512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:26.965512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:26.965664 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:26.965732 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:26.965940 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 03:28:26.966027 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 03:28:26.966140 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.931230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34729 (local address 127.0.0.1:48428)
0927 03:28:26.931390 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:26.931401 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:26.931475 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:26.931892 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:26.931893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:26.932115 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:26.932243 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:26.932245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.933215 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.933216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:26.934269 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:26.934271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.934791 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.934792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:26.934852 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.956491 (+ 21639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:26.956497 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:26.957816 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:26.964624 (+ 6808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:26.964625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:26.964626 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:26.964866 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:26.965268 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:26.965269 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:26.965269 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:26.965366 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:26.965766 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:26.965767 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:26.965840 (+ 73us) client_negotiation.cc:769] Sending connection context
0927 03:28:26.965915 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 03:28:26.966027 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":85,"thread_start_us":85,"threads_started":1}
I20260927 03:28:26.969000 7186 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:28:26.969099 7186 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:26.969197 7186 tablet_replica.cc:330] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896: stopping tablet replica
I20260927 03:28:26.969632 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39213
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41079
--webserver_port=44803
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:41079,127.0.98.61:34729
--tserver_master_addrs=127.0.98.62:41079,127.0.98.61:34729,127.0.98.60:40375 with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:28:26.969796 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.962265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34729 (local address 127.0.0.1:48442)
0927 03:28:26.963592 (+ 1327us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:26.963606 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:26.963659 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:26.966429 (+ 2770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:26.966431 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:26.966432 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:26.966479 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:26.966481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.967442 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.967443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:26.968606 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:26.968609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.969478 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.969479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:26.969549 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.969576 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 03:28:26.969632 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 03:28:26.969678 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1303}
I20260927 03:28:26.970083 7183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.962315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48442 (local address 127.0.98.61:34729)
0927 03:28:26.966214 (+ 3899us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:26.966216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:26.966224 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:28:26.966247 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:26.966248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:26.966275 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:26.966357 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:26.966553 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.967324 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.968721 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.969290 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.969365 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.969442 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:26.969969 (+ 527us) server_negotiation.cc:299] Negotiation successful
0927 03:28:26.970014 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3882,"thread_start_us":102,"threads_started":1}
I20260927 03:28:26.971413 7142 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [term 1 LEARNER]: Refusing update from remote peer abc21e02fdc0470ebb9a8fce09d9f081: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 03:28:26.971705 7280 consensus_queue.cc:1035] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 03:28:26.972302 7142 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 510a4c04062b4014b9466b0a818c5896 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: true } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: true } } }
I20260927 03:28:26.973502 7171 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abc21e02fdc0470ebb9a8fce09d9f081. Latest consensus state: current_term: 1 leader_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: true } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: true } } }
I20260927 03:28:26.973518 7284 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: true } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: true } } }
I20260927 03:28:26.973614 7171 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [sys.catalog]: This master's current role is: LEARNER
I20260927 03:28:26.973686 7284 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [sys.catalog]: This master's current role is: LEARNER
I20260927 03:28:26.974169 7280 raft_consensus.cc:1050] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081: attempting to promote NON_VOTER ab1b1713c6064c04a39dd6f981745533 to VOTER
I20260927 03:28:26.974402 7280 consensus_queue.cc:237] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: false } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: true } }
I20260927 03:28:26.974997 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.974848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40375 (local address 127.0.0.1:57372)
0927 03:28:26.974908 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:26.974958 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 03:28:26.975049 7142 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [term 1 LEARNER]: Refusing update from remote peer abc21e02fdc0470ebb9a8fce09d9f081: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 03:28:26.975250 7093 consensus_queue.cc:1035] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 03:28:26.975461 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.975325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40375 (local address 127.0.0.1:57378)
0927 03:28:26.975382 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:26.975406 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 03:28:26.975560 6517 consensus_peers.cc:487] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 -> Peer 510a4c04062b4014b9466b0a818c5896 (127.0.98.60:40375): Couldn't send request to peer 510a4c04062b4014b9466b0a818c5896. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 03:28:26.975984 7094 raft_consensus.cc:2932] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ab1b1713c6064c04a39dd6f981745533 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: false } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: true } } }
I20260927 03:28:26.976159 7142 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ab1b1713c6064c04a39dd6f981745533 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: false } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: true } } }
I20260927 03:28:26.977035 7094 sys_catalog.cc:455] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: false } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: true } } }
I20260927 03:28:26.977130 7094 sys_catalog.cc:458] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:26.977087 7093 sys_catalog.cc:455] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: false } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: true } } }
I20260927 03:28:26.977196 7093 sys_catalog.cc:458] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:26.977456 7284 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: false } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: true } } }
I20260927 03:28:26.977547 7284 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:28:26.983780 7186 master.cc:552] Master@127.0.98.60:40375 shutdown complete.
I20260927 03:28:26.983848 7186 master_runner.cc:305] Clearing existing system tablet
I20260927 03:28:26.984002 7186 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:26.984058 7186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:26.984864 7186 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:26.985493 7286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:26.985659 7186 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:26.985728 7186 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:26.985816 7186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-2/wal
uuid: "510a4c04062b4014b9466b0a818c5896"
format_stamp: "Formatted at 2026-09-27 03:28:26 on dist-test-slave-j0hf"
server_key: "7f4b334b5161e1a94057105e202bdb8a"
server_key_iv: "faf043e2a7ac7e52db8abe549254b9ce"
server_key_version: "encryptionkey@0"
I20260927 03:28:26.986665 7186 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:26.988169 7186 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 03:28:26.988252 7186 log.cc:1196] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 03:28:26.988615 7186 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896: Deleting consensus metadata
I20260927 03:28:26.988763 7186 master_runner.cc:315] Copying system tablet from 127.0.98.62:41079
I20260927 03:28:26.989418 7186 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:41079
I20260927 03:28:27.001601 7292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.994164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.0.1:45622)
0927 03:28:26.994357 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:26.994371 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:26.994458 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:26.994632 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:26.994634 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:26.994857 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:26.994934 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:26.994936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.995893 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.995894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:26.996792 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:26.996795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.997271 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.997272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:26.997344 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.997924 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:26.997929 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:26.998364 (+ 435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:27.000450 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:27.000451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:27.000451 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:27.000682 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:27.000916 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:27.000917 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:27.000917 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:27.000983 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:27.001274 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:27.001274 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:27.001324 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 03:28:27.001394 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 03:28:27.001498 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":68,"threads_started":1}
I20260927 03:28:27.001626 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:26.994259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45622 (local address 127.0.98.62:41079)
0927 03:28:26.994323 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:26.994326 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:26.994457 (+ 131us) server_negotiation.cc:407] Connection header received
0927 03:28:26.994534 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:26.994535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:26.994546 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:26.994610 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:26.995018 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.995800 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.996897 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:26.997181 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:26.997251 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:26.998550 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:26.998551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:26.998552 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:26.998560 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:27.000350 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:27.000770 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:27.000771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:27.000772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:27.000841 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:27.001052 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:27.001053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:27.001053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:27.001194 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:27.001274 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:27.001444 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 03:28:27.001527 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 03:28:27.002542 6553 tablet_copy_service.cc:140] P abc21e02fdc0470ebb9a8fce09d9f081: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 510a4c04062b4014b9466b0a818c5896 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:45622)
I20260927 03:28:27.002619 6553 tablet_copy_service.cc:161] P abc21e02fdc0470ebb9a8fce09d9f081: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 510a4c04062b4014b9466b0a818c5896 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:45622: session id = 510a4c04062b4014b9466b0a818c5896-00000000000000000000000000000000
I20260927 03:28:27.003218 6553 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 03:28:27.003744 7186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 03:28:27.005096 7186 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896: tablet copy: Starting download of 0 data blocks...
I20260927 03:28:27.005228 7186 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896: tablet copy: Starting download of 1 WAL segments...
I20260927 03:28:27.006136 7186 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 03:28:27.007107 6553 tablet_copy_service.cc:342] P abc21e02fdc0470ebb9a8fce09d9f081: Request end of tablet copy session 510a4c04062b4014b9466b0a818c5896-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:45622
I20260927 03:28:27.007176 6553 tablet_copy_service.cc:433] P abc21e02fdc0470ebb9a8fce09d9f081: ending tablet copy session 510a4c04062b4014b9466b0a818c5896-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 510a4c04062b4014b9466b0a818c5896
I20260927 03:28:27.009632 7186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:27.010797 7293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:27.012270 7294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:27.012460 7296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:27.012702 7186 server_base.cc:1034] running on GCE node
I20260927 03:28:27.012812 7186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:27.012959 7186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:27.014112 7186 hybrid_clock.cc:648] HybridClock initialized: now 1790479707014104 us; error 23 us; skew 500 ppm
Sep 27 03:28:27 dist-test-slave-j0hf krb5kdc[6488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479707, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:27.015746 7186 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 03:28:27.016034 7186 webserver.cc:460] Webserver started at http://127.0.98.60:35409/ using document root <none> and password file <none>
I20260927 03:28:27.016165 7186 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:27.016207 7186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:27.017047 7186 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:27.017728 7303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:27.017900 7186 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:27.017956 7186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-2/wal
uuid: "510a4c04062b4014b9466b0a818c5896"
format_stamp: "Formatted at 2026-09-27 03:28:26 on dist-test-slave-j0hf"
server_key: "7f4b334b5161e1a94057105e202bdb8a"
server_key_iv: "faf043e2a7ac7e52db8abe549254b9ce"
server_key_version: "encryptionkey@0"
I20260927 03:28:27.018029 7186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:27.025960 7186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:27.028015 7186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:27.031625 7186 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40375
I20260927 03:28:27.032023 7186 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/master-2/data/info.pb
I20260927 03:28:27.035873 7355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40375 every 8 connection(s)
I20260927 03:28:27.036391 7356 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:28:27.036880 7356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896: Bootstrap starting.
I20260927 03:28:27.039783 7356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:28:27.040002 7356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896: Bootstrap complete.
I20260927 03:28:27.040185 7356 raft_consensus.cc:348] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: false } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: true } }
I20260927 03:28:27.040280 7356 raft_consensus.cc:729] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 510a4c04062b4014b9466b0a818c5896, State: Initialized, Role: LEARNER
I20260927 03:28:27.040314 7356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: false } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: true } }
I20260927 03:28:27.040493 7356 sys_catalog.cc:564] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:28:27.041682 7359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: false } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: true } } }
I20260927 03:28:27.041950 7359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896 [sys.catalog]: This master's current role is: LEARNER
I20260927 03:28:27.042640 7370 catalog_manager.cc:1259] Loaded cluster ID: 0d21507db5a54eaaba302abd4d3f957a
I20260927 03:28:27.042696 7370 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896: loading cluster ID for follower catalog manager: success
I20260927 03:28:27.043864 7370 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896: acquiring CA information for follower catalog manager: success
I20260927 03:28:27.044179 7370 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 03:28:27.067853 7282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:27.068089 7282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:27.068130 7282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:27.069312 7282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:27.069379 7282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:27.069424 7282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:28:27.070941 7282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39213
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44803
--tserver_master_addrs=127.0.98.62:41079,127.0.98.61:34729,127.0.98.60:40375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:27.071218 7282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:27.071517 7282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:27.076566 7375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:27.076587 7376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:27.076994 7378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:27.077270 7282 server_base.cc:1034] running on GCE node
I20260927 03:28:27.077482 7282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:27.077768 7282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:27.078951 7282 hybrid_clock.cc:648] HybridClock initialized: now 1790479707078936 us; error 26 us; skew 500 ppm
Sep 27 03:28:27 dist-test-slave-j0hf krb5kdc[6488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479707, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:27.081647 7282 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 03:28:27.082201 7282 webserver.cc:460] Webserver started at http://127.0.98.1:44803/ using document root <none> and password file <none>
I20260927 03:28:27.082441 7282 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:27.082516 7282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:27.084062 7282 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:28:27.084770 7385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:27.085000 7282 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:27.085101 7282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "38ea6a029ea046ce8eae0d426bf8e485"
format_stamp: "Formatted at 2026-09-27 03:28:25 on dist-test-slave-j0hf"
server_key: "4a354d3de50d0711e06c5f9629284513"
server_key_iv: "3e32b2da851b33358d8220b77405e0d2"
server_key_version: "encryptionkey@0"
I20260927 03:28:27.085490 7282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:27.117432 7282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:27.120000 7282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:27.120167 7282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:27.120433 7282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:27.120748 7282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:27.120801 7282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:27.120847 7282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:27.120877 7282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:27.127326 7282 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39213
I20260927 03:28:27.127395 7498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39213 every 8 connection(s)
I20260927 03:28:27.127808 7282 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:28:27 dist-test-slave-j0hf krb5kdc[6488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479707, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 03:28:27 dist-test-slave-j0hf krb5kdc[6488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479707, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 03:28:27 dist-test-slave-j0hf krb5kdc[6488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479707, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 03:28:27.137573 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 7282
I20260927 03:28:27.137739 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 6729
I20260927 03:28:27.139221 7183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.128696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60525 (local address 127.0.98.61:34729)
0927 03:28:27.128784 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:27.128786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:27.129080 (+ 294us) server_negotiation.cc:407] Connection header received
0927 03:28:27.129718 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:27.129720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:27.129732 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:27.129768 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:27.130682 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.131452 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.132506 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.132788 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.132856 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.135145 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:27.135147 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:27.135147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:27.135154 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:27.137304 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:27.137988 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:27.137989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:27.137990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:27.138048 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:27.138348 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:27.138349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:27.138349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:27.138516 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:27.138561 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:27.138983 (+ 422us) server_negotiation.cc:299] Negotiation successful
0927 03:28:27.139073 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 03:28:27.139250 7505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.128728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39887 (local address 127.0.98.60:40375)
0927 03:28:27.128845 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:27.128847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:27.129080 (+ 233us) server_negotiation.cc:407] Connection header received
0927 03:28:27.129719 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:27.129721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:27.129733 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:27.129788 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:27.130682 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.131706 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.132649 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.132890 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.132959 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.136299 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:27.136301 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:27.136301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:27.136310 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:27.137971 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:27.138321 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:27.138322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:27.138323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:27.138394 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:27.138609 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:27.138609 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:27.138610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:27.138768 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:27.138811 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:27.139031 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 03:28:27.139125 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":46,"threads_started":1}
I20260927 03:28:27.139472 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.128635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42205 (local address 127.0.98.62:41079)
0927 03:28:27.128746 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:27.128748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:27.129094 (+ 346us) server_negotiation.cc:407] Connection header received
0927 03:28:27.129701 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:27.129702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:27.129715 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:27.129757 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:27.130672 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.131172 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.132369 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.132634 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.132696 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.135638 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:27.135640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:27.135641 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:27.135650 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:27.137360 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:27.138632 (+ 1272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:27.138633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:27.138633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:27.138686 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:27.138888 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:27.138889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:27.138890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:27.139027 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:27.139069 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:27.139269 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 03:28:27.139363 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 03:28:27.139885 7503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.128605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.98.1:42205)
0927 03:28:27.128929 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:27.128995 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:27.129606 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:27.130156 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:27.130157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:27.130368 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:27.130571 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:27.130575 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.131282 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.131283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:27.132277 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:27.132279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.132762 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.132763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:27.132977 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.133724 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:27.133730 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:27.135518 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:27.138334 (+ 2816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:27.138335 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:27.138335 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:27.138546 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:27.138761 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:27.138761 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:27.138761 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:27.138815 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:27.139115 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:27.139116 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:27.139161 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 03:28:27.139613 (+ 452us) client_negotiation.cc:240] Negotiation successful
0927 03:28:27.139699 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":87,"threads_started":1}
I20260927 03:28:27.139890 7504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.128605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40375 (local address 127.0.98.1:39887)
0927 03:28:27.128958 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:27.128997 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:27.129597 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:27.129866 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:27.129867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:27.130396 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:27.130571 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:27.130577 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.131797 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.131798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:27.132559 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:27.132562 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.132964 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.132964 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:27.133017 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.133852 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:27.133856 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:27.136171 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:27.138054 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:27.138055 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:27.138056 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:27.138258 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:27.138466 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:27.138467 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:27.138467 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:27.138530 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:27.138871 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:27.138872 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:27.138915 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 03:28:27.139327 (+ 412us) client_negotiation.cc:240] Negotiation successful
0927 03:28:27.139646 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":358,"thread_start_us":63,"threads_started":1}
I20260927 03:28:27.139891 7506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.128605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34729 (local address 127.0.98.1:60525)
0927 03:28:27.128930 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:27.129012 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:27.129597 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:27.129852 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:27.129857 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:27.130392 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:27.130571 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:27.130577 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.131540 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.131541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:27.132427 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:27.132429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.132865 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.132866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:27.132978 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.133502 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:27.133511 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:27.135050 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:27.137528 (+ 2478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:27.137538 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:27.137541 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:27.137882 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:27.138118 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:27.138119 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:27.138119 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:27.138273 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:27.138606 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:27.138612 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:27.138803 (+ 191us) client_negotiation.cc:769] Sending connection context
0927 03:28:27.139313 (+ 510us) client_negotiation.cc:240] Negotiation successful
0927 03:28:27.139561 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":15,"thread_start_us":93,"threads_started":1}
I20260927 03:28:27.140617 7499 heartbeater.cc:347] Connected to a master server at 127.0.98.60:40375
I20260927 03:28:27.140715 7499 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:27.140864 7499 heartbeater.cc:511] Master 127.0.98.60:40375 requested a full tablet report, sending...
I20260927 03:28:27.141407 7314 ts_manager.cc:194] Registered new tserver with Master: 38ea6a029ea046ce8eae0d426bf8e485 (127.0.98.1:39213)
I20260927 03:28:27.143581 7501 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34729
I20260927 03:28:27.143636 7501 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:27.143692 7501 heartbeater.cc:511] Master 127.0.98.61:34729 requested a full tablet report, sending...
I20260927 03:28:27.144085 7132 ts_manager.cc:194] Registered new tserver with Master: 38ea6a029ea046ce8eae0d426bf8e485 (127.0.98.1:39213)
I20260927 03:28:27.144887 7500 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41079
I20260927 03:28:27.144953 7500 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:27.145023 7500 heartbeater.cc:511] Master 127.0.98.62:41079 requested a full tablet report, sending...
I20260927 03:28:27.145148 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44295
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41079
--webserver_port=43127
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:41079,127.0.98.61:34729
--tserver_master_addrs=127.0.98.62:41079,127.0.98.61:34729,127.0.98.60:40375 with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:28:27.145294 6533 ts_manager.cc:194] Re-registered known tserver with Master: 38ea6a029ea046ce8eae0d426bf8e485 (127.0.98.1:39213)
I20260927 03:28:27.145684 6533 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42205
W20260927 03:28:27.227670 7507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:27.227869 7507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:27.227892 7507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:27.229315 7507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:27.229362 7507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:27.229388 7507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:28:27.231014 7507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44295
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43127
--tserver_master_addrs=127.0.98.62:41079,127.0.98.61:34729,127.0.98.60:40375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:27.231221 7507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:27.231392 7507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:27.236441 7512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:27.236452 7513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:27.236663 7507 server_base.cc:1034] running on GCE node
W20260927 03:28:27.236822 7515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:27.237095 7507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:27.237323 7507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:27.238473 7507 hybrid_clock.cc:648] HybridClock initialized: now 1790479707238456 us; error 45 us; skew 500 ppm
Sep 27 03:28:27 dist-test-slave-j0hf krb5kdc[6488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479707, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:27.241144 7507 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 03:28:27.241680 7507 webserver.cc:460] Webserver started at http://127.0.98.2:43127/ using document root <none> and password file <none>
I20260927 03:28:27.241986 7507 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:27.242076 7507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:27.243533 7507 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:28:27.244158 7522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:27.244334 7507 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:27.244431 7507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "adc03843909b4012b156274ce3ca2535"
format_stamp: "Formatted at 2026-09-27 03:28:25 on dist-test-slave-j0hf"
server_key: "e396ed988ea0f9b2ade731cc3fa9fb69"
server_key_iv: "f10bbae4e293338d1e493c79b4170b07"
server_key_version: "encryptionkey@0"
I20260927 03:28:27.244817 7507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:27.255852 7507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:27.257711 7507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:27.257869 7507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:27.258080 7507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:27.258369 7507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:27.258422 7507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:27.258471 7507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:27.258517 7507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:27.264367 7507 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44295
I20260927 03:28:27.264441 7635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44295 every 8 connection(s)
I20260927 03:28:27.264789 7507 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:28:27.270160 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 7507
I20260927 03:28:27.271076 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 6863
Sep 27 03:28:27 dist-test-slave-j0hf krb5kdc[6488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479707, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 03:28:27 dist-test-slave-j0hf krb5kdc[6488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479707, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 03:28:27.277083 7183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.265745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52929 (local address 127.0.98.61:34729)
0927 03:28:27.265817 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:27.265819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:27.265990 (+ 171us) server_negotiation.cc:407] Connection header received
0927 03:28:27.267123 (+ 1133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:27.267124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:27.267136 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:27.267184 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:27.267965 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.268718 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.269846 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.270153 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.270221 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.273387 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:27.273388 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:27.273389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:27.273396 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:27.275561 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:27.276009 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:27.276010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:27.276011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:27.276086 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:27.276381 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:27.276383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:27.276383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:27.276534 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:27.276579 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:27.276867 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 03:28:27.276974 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 03:28:27.277117 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.265790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43107 (local address 127.0.98.62:41079)
0927 03:28:27.265840 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:27.265842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:27.266148 (+ 306us) server_negotiation.cc:407] Connection header received
0927 03:28:27.267097 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:27.267098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:27.267110 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:27.267141 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:27.267950 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.268474 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.269561 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.269803 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.269868 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.273453 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:27.273455 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:27.273455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:27.273462 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:27.275385 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:27.276050 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:27.276051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:27.276051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:27.276116 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:27.276369 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:27.276370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:27.276370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:27.276498 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:27.276574 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:27.276903 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 03:28:27.276986 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:28:27.277307 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34339
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41079
--webserver_port=45149
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:41079,127.0.98.61:34729
--tserver_master_addrs=127.0.98.62:41079,127.0.98.61:34729,127.0.98.60:40375 with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:28:27.277688 7641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.265510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34729 (local address 127.0.98.2:52929)
0927 03:28:27.265898 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:27.265906 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:27.266943 (+ 1037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:27.267225 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:27.267226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:27.267700 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:27.267855 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:27.267868 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.268824 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.268825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:27.269741 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:27.269743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.270284 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.270285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:27.270353 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.270941 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:27.270946 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:27.273273 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:27.275636 (+ 2363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:27.275636 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:27.275637 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:27.275929 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:27.276160 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:27.276160 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:27.276161 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:27.276298 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:27.276626 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:27.276626 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:27.276722 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 03:28:27.277181 (+ 459us) client_negotiation.cc:240] Negotiation successful
0927 03:28:27.277412 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":71,"thread_start_us":72,"threads_started":1}
I20260927 03:28:27.277695 7642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.265509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.98.2:43107)
0927 03:28:27.266098 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:27.266107 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:27.266959 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:27.267218 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:27.267218 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:27.267661 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:27.267854 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:27.267859 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.268587 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.268587 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:27.269456 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:27.269458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.269953 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.269954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:27.270096 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.270678 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:27.270690 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:27.273317 (+ 2627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:27.275506 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:27.275513 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:27.275516 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:27.275931 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:27.276197 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:27.276198 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:27.276198 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:27.276289 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:27.276565 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:27.276571 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:27.276720 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 03:28:27.277243 (+ 523us) client_negotiation.cc:240] Negotiation successful
0927 03:28:27.277498 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":127,"thread_start_us":86,"threads_started":1}
Sep 27 03:28:27 dist-test-slave-j0hf krb5kdc[6488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479707, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 03:28:27.278245 7637 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41079
I20260927 03:28:27.278345 7637 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:27.278546 7637 heartbeater.cc:511] Master 127.0.98.62:41079 requested a full tablet report, sending...
I20260927 03:28:27.279006 6533 ts_manager.cc:194] Re-registered known tserver with Master: adc03843909b4012b156274ce3ca2535 (127.0.98.2:44295)
I20260927 03:28:27.279736 7638 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34729
I20260927 03:28:27.279855 7638 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:27.279860 6533 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43107
I20260927 03:28:27.279976 7638 heartbeater.cc:511] Master 127.0.98.61:34729 requested a full tablet report, sending...
I20260927 03:28:27.280323 7132 ts_manager.cc:194] Registered new tserver with Master: adc03843909b4012b156274ce3ca2535 (127.0.98.2:44295)
I20260927 03:28:27.281279 7505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.265797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48791 (local address 127.0.98.60:40375)
0927 03:28:27.265841 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:27.265843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:27.265990 (+ 147us) server_negotiation.cc:407] Connection header received
0927 03:28:27.267054 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:27.267055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:27.267068 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:27.267126 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:27.267966 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.268765 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.269828 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.270104 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.270166 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.278392 (+ 8226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:27.278394 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:27.278394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:27.278405 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:27.280111 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:27.280467 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:27.280469 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:27.280469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:27.280528 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:27.280748 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:27.280749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:27.280749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:27.280863 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:27.280938 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:27.281086 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 03:28:27.281178 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:28:27.281508 7640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.265512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40375 (local address 127.0.98.2:48791)
0927 03:28:27.265830 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:27.265887 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:27.266943 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:27.267197 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:27.267201 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:27.267692 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:27.267854 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:27.267860 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.268879 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.268881 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:27.269741 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:27.269743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.270217 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.270217 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:27.270285 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.270750 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:27.270754 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:27.278284 (+ 7530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:27.280201 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:27.280201 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:27.280202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:27.280373 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:27.280608 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:27.280609 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:27.280609 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:27.280672 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:27.280930 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:27.280931 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:27.280973 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 03:28:27.281063 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 03:28:27.281177 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":22,"thread_start_us":86,"threads_started":1}
I20260927 03:28:27.281872 7636 heartbeater.cc:347] Connected to a master server at 127.0.98.60:40375
I20260927 03:28:27.281948 7636 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:27.282027 7636 heartbeater.cc:511] Master 127.0.98.60:40375 requested a full tablet report, sending...
I20260927 03:28:27.282335 7314 ts_manager.cc:194] Registered new tserver with Master: adc03843909b4012b156274ce3ca2535 (127.0.98.2:44295)
W20260927 03:28:27.362702 7643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:27.362907 7643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:27.362931 7643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:27.364106 7643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:27.364148 7643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:27.364174 7643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:28:27.365692 7643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34339
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45149
--tserver_master_addrs=127.0.98.62:41079,127.0.98.61:34729,127.0.98.60:40375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:27.365942 7643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:27.366186 7643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:27.371367 7649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:27.371404 7651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:27.371377 7648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:27.371608 7643 server_base.cc:1034] running on GCE node
I20260927 03:28:27.371883 7643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:27.372148 7643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:27.373318 7643 hybrid_clock.cc:648] HybridClock initialized: now 1790479707373312 us; error 37 us; skew 500 ppm
Sep 27 03:28:27 dist-test-slave-j0hf krb5kdc[6488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479707, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:27.375936 7643 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 03:28:27.376397 7643 webserver.cc:460] Webserver started at http://127.0.98.3:45149/ using document root <none> and password file <none>
I20260927 03:28:27.376623 7643 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:27.376681 7643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:27.378038 7643 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:27.378626 7658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:27.378821 7643 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:27.378923 7643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "972f0abeb34849df82b4eca3335ec62d"
format_stamp: "Formatted at 2026-09-27 03:28:26 on dist-test-slave-j0hf"
server_key: "0a162aa99db5bc5f7dbe570217d6f1b7"
server_key_iv: "8975cebbd3a2a5456edde2a6e457f6e6"
server_key_version: "encryptionkey@0"
I20260927 03:28:27.379269 7643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:27.395253 7643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:27.397694 7643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:27.397873 7643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:27.398120 7643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:27.398510 7643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:27.398564 7643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:27.398607 7643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:27.398633 7643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:27.404565 7643 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34339
I20260927 03:28:27.404613 7771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34339 every 8 connection(s)
I20260927 03:28:27.404951 7643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:28:27 dist-test-slave-j0hf krb5kdc[6488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479707, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 03:28:27 dist-test-slave-j0hf krb5kdc[6488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479707, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 03:28:27 dist-test-slave-j0hf krb5kdc[6488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479707, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 03:28:27.415045 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 7643
I20260927 03:28:27.416217 7183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.405703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58247 (local address 127.0.98.61:34729)
0927 03:28:27.405750 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:27.405754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:27.406070 (+ 316us) server_negotiation.cc:407] Connection header received
0927 03:28:27.407184 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:27.407185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:27.407198 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:27.407232 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:27.408023 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.408515 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.409698 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.409965 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.410008 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.412823 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:27.412824 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:27.412825 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:27.412833 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:27.414505 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:27.415039 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:27.415040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:27.415040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:27.415106 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:27.415416 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:27.415417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:27.415417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:27.415613 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:27.415702 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:27.416027 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 03:28:27.416111 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:28:27.416494 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.405709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34729 (local address 127.0.98.3:58247)
0927 03:28:27.405913 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:27.405948 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:27.407056 (+ 1108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:27.407329 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:27.407334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:27.407719 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:27.407921 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:27.407925 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.408623 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.408624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:27.409578 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:27.409581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.410094 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.410095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:27.410255 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.410899 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:27.410910 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:27.412736 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:27.414595 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:27.414602 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:27.414605 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:27.414913 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:27.415196 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:27.415197 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:27.415197 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:27.415344 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:27.415691 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:27.415697 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:27.415872 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 03:28:27.416002 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 03:28:27.416200 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":119,"threads_started":1}
I20260927 03:28:27.416501 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.405709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.98.3:35459)
0927 03:28:27.406155 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:27.406164 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:27.407056 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:27.407329 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:27.407334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:27.407750 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:27.407922 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:27.407928 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.408910 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.408911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:27.409845 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:27.409847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.410700 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.410701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:27.410770 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.411213 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:27.411217 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:27.412991 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:27.414904 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:27.414905 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:27.414905 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:27.415101 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:27.415304 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:27.415305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:27.415305 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:27.415361 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:27.415840 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:27.415841 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:27.415886 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 03:28:27.416002 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 03:28:27.416276 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":176,"spinlock_wait_cycles":8576,"thread_start_us":113,"threads_started":1}
I20260927 03:28:27.416950 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.406002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35459 (local address 127.0.98.62:41079)
0927 03:28:27.406055 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:27.406057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:27.406212 (+ 155us) server_negotiation.cc:407] Connection header received
0927 03:28:27.407184 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:27.407185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:27.407198 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:27.407232 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:27.408032 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.408824 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.410347 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.410608 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.410688 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.413106 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:27.413107 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:27.413107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:27.413115 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:27.414816 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:27.415172 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:27.415173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:27.415174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:27.415228 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:27.415572 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:27.415574 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:27.415574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:27.415735 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:27.415809 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:27.416761 (+ 952us) server_negotiation.cc:299] Negotiation successful
0927 03:28:27.416845 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:28:27.417212 7774 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34729
I20260927 03:28:27.417253 7773 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41079
I20260927 03:28:27.417313 7774 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:27.417313 7773 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:27.417534 7774 heartbeater.cc:511] Master 127.0.98.61:34729 requested a full tablet report, sending...
I20260927 03:28:27.417534 7773 heartbeater.cc:511] Master 127.0.98.62:41079 requested a full tablet report, sending...
I20260927 03:28:27.417912 6533 ts_manager.cc:194] Re-registered known tserver with Master: 972f0abeb34849df82b4eca3335ec62d (127.0.98.3:34339)
I20260927 03:28:27.418011 7132 ts_manager.cc:194] Registered new tserver with Master: 972f0abeb34849df82b4eca3335ec62d (127.0.98.3:34339)
I20260927 03:28:27.418313 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.406047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40375 (local address 127.0.98.3:37569)
0927 03:28:27.406203 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:27.406214 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:27.407072 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:27.407329 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:27.407334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:27.407758 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:27.407922 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:27.407928 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.408896 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.408897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:27.409845 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:27.409847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.410322 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.410324 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:27.410398 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.411054 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:27.411058 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:27.413604 (+ 2546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:27.416886 (+ 3282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:27.416887 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:27.416887 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:27.417090 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:27.417313 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:27.417314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:27.417314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:27.417370 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:27.417683 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:27.417684 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:27.417740 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 03:28:27.417806 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 03:28:27.417907 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":33,"thread_start_us":65,"threads_started":1}
I20260927 03:28:27.418433 6533 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35459
I20260927 03:28:27.418627 7505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.406167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37569 (local address 127.0.98.60:40375)
0927 03:28:27.406210 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:27.406212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:27.406270 (+ 58us) server_negotiation.cc:407] Connection header received
0927 03:28:27.407168 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:27.407169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:27.407183 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:27.407237 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:27.408032 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.408793 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.409929 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.410218 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.410287 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.413691 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:27.413692 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:27.413692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:27.413698 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:27.416801 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:27.417178 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:27.417179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:27.417179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:27.417242 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:27.417441 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:27.417442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:27.417442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:27.417590 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:27.417655 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:27.418420 (+ 765us) server_negotiation.cc:299] Negotiation successful
0927 03:28:27.418521 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:28:27.418965 7772 heartbeater.cc:347] Connected to a master server at 127.0.98.60:40375
I20260927 03:28:27.419026 7772 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:27.419142 7772 heartbeater.cc:511] Master 127.0.98.60:40375 requested a full tablet report, sending...
I20260927 03:28:27.419515 7314 ts_manager.cc:194] Registered new tserver with Master: 972f0abeb34849df82b4eca3335ec62d (127.0.98.3:34339)
I20260927 03:28:27.427027 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.417738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45636 (local address 127.0.98.62:41079)
0927 03:28:27.418904 (+ 1166us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:27.418906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:27.419518 (+ 612us) server_negotiation.cc:407] Connection header received
0927 03:28:27.419590 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:27.419591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:27.419601 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:27.419648 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:27.420493 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.421195 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.421902 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.422171 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.422234 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.424154 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:27.424155 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:27.424155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:27.424163 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:27.425702 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:27.426187 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:27.426188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:27.426189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:27.426241 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:27.426479 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:27.426480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:27.426480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:27.426586 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:27.426696 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:27.426870 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 03:28:27.426953 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1139}
I20260927 03:28:27.427016 7183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.417553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48454 (local address 127.0.98.61:34729)
0927 03:28:27.417589 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:27.417591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:27.419625 (+ 2034us) server_negotiation.cc:407] Connection header received
0927 03:28:27.419688 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:27.419689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:27.419699 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:27.419749 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:27.420526 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.420990 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.422045 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.422308 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.422350 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.424002 (+ 1652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:27.424003 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:27.424004 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:27.424011 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:27.425685 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:27.426171 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:27.426172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:27.426172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:27.426225 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:27.426426 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:27.426426 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:27.426427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:27.426571 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:27.426669 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:27.426808 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 03:28:27.426894 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:28:27.427260 7505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.419117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57388 (local address 127.0.98.60:40375)
0927 03:28:27.419156 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:27.419159 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:27.419774 (+ 615us) server_negotiation.cc:407] Connection header received
0927 03:28:27.419833 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:27.419834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:27.419843 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:27.419898 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:27.420411 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.421145 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.422078 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.422349 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.422425 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.423947 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:27.423949 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:27.423949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:27.423963 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:27.425963 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:27.426347 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:27.426348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:27.426348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:27.426398 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:27.426599 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:27.426599 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:27.426600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:27.426729 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:27.426799 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:27.427111 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 03:28:27.427190 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:28:27.428982 6533 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45636:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:28:27.430686 6533 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:28:27.438344 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.434423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60892 (local address 127.0.98.3:34339)
0927 03:28:27.434590 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:27.434596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:27.435067 (+ 471us) server_negotiation.cc:407] Connection header received
0927 03:28:27.435107 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:27.435109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:27.435125 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:27.435197 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:27.435308 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.436374 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.437387 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.437968 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.438078 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.438161 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:27.438226 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 03:28:27.438262 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
I20260927 03:28:27.438351 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.434195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39213 (local address 127.0.0.1:54694)
0927 03:28:27.434228 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:27.434240 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:27.434277 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:27.434954 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:27.434955 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:27.434973 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:27.435003 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:27.435004 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.436048 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.436049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:27.437175 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:27.437178 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.438120 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.438121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:27.438183 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.438187 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:28:27.438218 (+ 31us) client_negotiation.cc:240] Negotiation successful
0927 03:28:27.438260 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260927 03:28:27.438344 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.434288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34339 (local address 127.0.0.1:60892)
0927 03:28:27.434931 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:27.434939 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:27.435007 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:27.435210 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:27.435211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:27.435213 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:27.435246 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:27.435248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.436464 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.436465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:27.437304 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:27.437306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.438080 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.438081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:27.438144 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.438150 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 03:28:27.438212 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 03:28:27.438252 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":84,"threads_started":1}
I20260927 03:28:27.438778 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.434261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44295 (local address 127.0.0.1:59478)
0927 03:28:27.434389 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:27.434401 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:27.434650 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:27.434868 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:27.434869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:27.434872 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:27.434929 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:27.434931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.436245 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.436246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:27.437452 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:27.437454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.438322 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.438322 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:27.438381 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.438384 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:28:27.438413 (+ 29us) client_negotiation.cc:240] Negotiation successful
0927 03:28:27.438695 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":82,"thread_start_us":74,"threads_started":1}
I20260927 03:28:27.438925 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.434534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54694 (local address 127.0.98.1:39213)
0927 03:28:27.434776 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:27.434780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:27.434788 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:28:27.434819 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:27.434821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:27.434855 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:27.435074 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:27.435118 (+ 44us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.435951 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.437262 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.438020 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.438595 (+ 575us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.438739 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:27.438773 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 03:28:27.438820 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":74,"threads_started":1}
I20260927 03:28:27.439129 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.434373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59478 (local address 127.0.98.2:44295)
0927 03:28:27.434710 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:27.434714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:27.434723 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:28:27.434762 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:27.434770 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:27.434783 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:27.434856 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:27.435011 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.436166 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.437536 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.438234 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.438899 (+ 665us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.438985 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:27.439012 (+ 27us) server_negotiation.cc:299] Negotiation successful
0927 03:28:27.439057 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":77,"threads_started":1}
I20260927 03:28:27.439255 7706 tablet_service.cc:1467] Processing CreateTablet for tablet 57792c0254fe43a0ab2b275272afaab2 (DEFAULT_TABLE table=test-table [id=db474b35b6da46fa84d19b5951c56f9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:27.439687 7706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57792c0254fe43a0ab2b275272afaab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:27.439591 7433 tablet_service.cc:1467] Processing CreateTablet for tablet 57792c0254fe43a0ab2b275272afaab2 (DEFAULT_TABLE table=test-table [id=db474b35b6da46fa84d19b5951c56f9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:27.439659 7570 tablet_service.cc:1467] Processing CreateTablet for tablet 57792c0254fe43a0ab2b275272afaab2 (DEFAULT_TABLE table=test-table [id=db474b35b6da46fa84d19b5951c56f9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:27.439949 7433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57792c0254fe43a0ab2b275272afaab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:27.439949 7570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57792c0254fe43a0ab2b275272afaab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:27.442982 7795 tablet_bootstrap.cc:492] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d: Bootstrap starting.
I20260927 03:28:27.443289 7796 tablet_bootstrap.cc:492] T 57792c0254fe43a0ab2b275272afaab2 P 38ea6a029ea046ce8eae0d426bf8e485: Bootstrap starting.
I20260927 03:28:27.443548 7797 tablet_bootstrap.cc:492] T 57792c0254fe43a0ab2b275272afaab2 P adc03843909b4012b156274ce3ca2535: Bootstrap starting.
I20260927 03:28:27.444079 7797 tablet_bootstrap.cc:654] T 57792c0254fe43a0ab2b275272afaab2 P adc03843909b4012b156274ce3ca2535: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:27.444296 7795 tablet_bootstrap.cc:654] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:27.444365 7796 tablet_bootstrap.cc:654] T 57792c0254fe43a0ab2b275272afaab2 P 38ea6a029ea046ce8eae0d426bf8e485: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:27.444396 7797 log.cc:824] T 57792c0254fe43a0ab2b275272afaab2 P adc03843909b4012b156274ce3ca2535: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:27.444613 7795 log.cc:824] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:27.444643 7796 log.cc:824] T 57792c0254fe43a0ab2b275272afaab2 P 38ea6a029ea046ce8eae0d426bf8e485: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:27.445116 7797 tablet_bootstrap.cc:492] T 57792c0254fe43a0ab2b275272afaab2 P adc03843909b4012b156274ce3ca2535: No bootstrap required, opened a new log
I20260927 03:28:27.445170 7797 ts_tablet_manager.cc:1397] T 57792c0254fe43a0ab2b275272afaab2 P adc03843909b4012b156274ce3ca2535: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:28:27.446435 7797 raft_consensus.cc:348] T 57792c0254fe43a0ab2b275272afaab2 P adc03843909b4012b156274ce3ca2535 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea6a029ea046ce8eae0d426bf8e485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39213 } } peers { permanent_uuid: "adc03843909b4012b156274ce3ca2535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "972f0abeb34849df82b4eca3335ec62d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34339 } }
I20260927 03:28:27.446560 7797 raft_consensus.cc:374] T 57792c0254fe43a0ab2b275272afaab2 P adc03843909b4012b156274ce3ca2535 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:27.446581 7797 raft_consensus.cc:729] T 57792c0254fe43a0ab2b275272afaab2 P adc03843909b4012b156274ce3ca2535 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adc03843909b4012b156274ce3ca2535, State: Initialized, Role: FOLLOWER
I20260927 03:28:27.446723 7797 consensus_queue.cc:260] T 57792c0254fe43a0ab2b275272afaab2 P adc03843909b4012b156274ce3ca2535 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea6a029ea046ce8eae0d426bf8e485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39213 } } peers { permanent_uuid: "adc03843909b4012b156274ce3ca2535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "972f0abeb34849df82b4eca3335ec62d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34339 } }
I20260927 03:28:27.447012 7797 ts_tablet_manager.cc:1428] T 57792c0254fe43a0ab2b275272afaab2 P adc03843909b4012b156274ce3ca2535: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:28:27.447041 7637 heartbeater.cc:503] Master 127.0.98.62:41079 was elected leader, sending a full tablet report...
I20260927 03:28:27.447911 7796 tablet_bootstrap.cc:492] T 57792c0254fe43a0ab2b275272afaab2 P 38ea6a029ea046ce8eae0d426bf8e485: No bootstrap required, opened a new log
I20260927 03:28:27.447979 7796 ts_tablet_manager.cc:1397] T 57792c0254fe43a0ab2b275272afaab2 P 38ea6a029ea046ce8eae0d426bf8e485: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 03:28:27.450138 7796 raft_consensus.cc:348] T 57792c0254fe43a0ab2b275272afaab2 P 38ea6a029ea046ce8eae0d426bf8e485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea6a029ea046ce8eae0d426bf8e485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39213 } } peers { permanent_uuid: "adc03843909b4012b156274ce3ca2535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "972f0abeb34849df82b4eca3335ec62d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34339 } }
I20260927 03:28:27.450294 7796 raft_consensus.cc:374] T 57792c0254fe43a0ab2b275272afaab2 P 38ea6a029ea046ce8eae0d426bf8e485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:27.450338 7796 raft_consensus.cc:729] T 57792c0254fe43a0ab2b275272afaab2 P 38ea6a029ea046ce8eae0d426bf8e485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ea6a029ea046ce8eae0d426bf8e485, State: Initialized, Role: FOLLOWER
I20260927 03:28:27.450452 7796 consensus_queue.cc:260] T 57792c0254fe43a0ab2b275272afaab2 P 38ea6a029ea046ce8eae0d426bf8e485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea6a029ea046ce8eae0d426bf8e485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39213 } } peers { permanent_uuid: "adc03843909b4012b156274ce3ca2535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "972f0abeb34849df82b4eca3335ec62d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34339 } }
I20260927 03:28:27.450762 7796 ts_tablet_manager.cc:1428] T 57792c0254fe43a0ab2b275272afaab2 P 38ea6a029ea046ce8eae0d426bf8e485: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:28:27.450826 7500 heartbeater.cc:503] Master 127.0.98.62:41079 was elected leader, sending a full tablet report...
I20260927 03:28:27.450982 7795 tablet_bootstrap.cc:492] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d: No bootstrap required, opened a new log
I20260927 03:28:27.451053 7795 ts_tablet_manager.cc:1397] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20260927 03:28:27.453132 7795 raft_consensus.cc:348] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea6a029ea046ce8eae0d426bf8e485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39213 } } peers { permanent_uuid: "adc03843909b4012b156274ce3ca2535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "972f0abeb34849df82b4eca3335ec62d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34339 } }
I20260927 03:28:27.453277 7795 raft_consensus.cc:374] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:27.453322 7795 raft_consensus.cc:729] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 972f0abeb34849df82b4eca3335ec62d, State: Initialized, Role: FOLLOWER
I20260927 03:28:27.453438 7795 consensus_queue.cc:260] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea6a029ea046ce8eae0d426bf8e485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39213 } } peers { permanent_uuid: "adc03843909b4012b156274ce3ca2535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "972f0abeb34849df82b4eca3335ec62d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34339 } }
I20260927 03:28:27.453722 7795 ts_tablet_manager.cc:1428] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:28:27.453754 7773 heartbeater.cc:503] Master 127.0.98.62:41079 was elected leader, sending a full tablet report...
I20260927 03:28:27.457710 7803 raft_consensus.cc:482] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:28:27.457823 7803 raft_consensus.cc:504] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea6a029ea046ce8eae0d426bf8e485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39213 } } peers { permanent_uuid: "adc03843909b4012b156274ce3ca2535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "972f0abeb34849df82b4eca3335ec62d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34339 } }
I20260927 03:28:27.458102 7803 leader_election.cc:288] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38ea6a029ea046ce8eae0d426bf8e485 (127.0.98.1:39213), adc03843909b4012b156274ce3ca2535 (127.0.98.2:44295)
I20260927 03:28:27.461932 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.458368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39173 (local address 127.0.98.2:44295)
0927 03:28:27.458406 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:27.458407 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:27.458415 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:28:27.458487 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:27.458488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:27.458499 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:27.458552 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:27.458701 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.459392 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.460927 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.461518 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.461621 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.461633 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:27.461786 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 03:28:27.461876 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:28:27.461933 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.458271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44295 (local address 127.0.98.3:39173)
0927 03:28:27.458327 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:27.458357 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:27.458404 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:27.458568 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:27.458569 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:27.458570 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:27.458634 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:27.458636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.459582 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.459583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:27.460832 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:27.460834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.461621 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.461622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:27.461702 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.461708 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 03:28:27.461780 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 03:28:27.461822 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":37}
I20260927 03:28:27.462129 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.458221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39213 (local address 127.0.98.3:47271)
0927 03:28:27.458286 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:27.458298 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:27.458337 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:27.458505 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:27.458506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:27.458508 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:27.458581 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:27.458582 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.459509 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.459510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:27.460831 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:27.460834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.461866 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.461867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:27.461938 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.461942 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:28:27.462008 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 03:28:27.462049 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":40}
I20260927 03:28:27.462404 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.458314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47271 (local address 127.0.98.1:39213)
0927 03:28:27.458379 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:27.458381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:27.458391 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:28:27.458416 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:27.458417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:27.458428 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:27.458491 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:27.458645 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.459392 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.461212 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.461747 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.461821 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.461831 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:27.462288 (+ 457us) server_negotiation.cc:299] Negotiation successful
0927 03:28:27.462334 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 03:28:27.462718 7453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57792c0254fe43a0ab2b275272afaab2" candidate_uuid: "972f0abeb34849df82b4eca3335ec62d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ea6a029ea046ce8eae0d426bf8e485" is_pre_election: true
I20260927 03:28:27.462869 7453 raft_consensus.cc:2454] T 57792c0254fe43a0ab2b275272afaab2 P 38ea6a029ea046ce8eae0d426bf8e485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 972f0abeb34849df82b4eca3335ec62d in term 0.
I20260927 03:28:27.463125 7661 leader_election.cc:302] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38ea6a029ea046ce8eae0d426bf8e485, 972f0abeb34849df82b4eca3335ec62d; no voters:
I20260927 03:28:27.463263 7803 raft_consensus.cc:2781] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:28:27.463323 7803 raft_consensus.cc:482] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:28:27.463354 7803 raft_consensus.cc:3037] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:27.463650 7590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57792c0254fe43a0ab2b275272afaab2" candidate_uuid: "972f0abeb34849df82b4eca3335ec62d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc03843909b4012b156274ce3ca2535" is_pre_election: true
I20260927 03:28:27.463800 7590 raft_consensus.cc:2454] T 57792c0254fe43a0ab2b275272afaab2 P adc03843909b4012b156274ce3ca2535 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 972f0abeb34849df82b4eca3335ec62d in term 0.
I20260927 03:28:27.464370 7803 raft_consensus.cc:504] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea6a029ea046ce8eae0d426bf8e485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39213 } } peers { permanent_uuid: "adc03843909b4012b156274ce3ca2535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "972f0abeb34849df82b4eca3335ec62d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34339 } }
I20260927 03:28:27.464617 7803 leader_election.cc:288] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d [CANDIDATE]: Term 1 election: Requested vote from peers 38ea6a029ea046ce8eae0d426bf8e485 (127.0.98.1:39213), adc03843909b4012b156274ce3ca2535 (127.0.98.2:44295)
I20260927 03:28:27.464694 7453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57792c0254fe43a0ab2b275272afaab2" candidate_uuid: "972f0abeb34849df82b4eca3335ec62d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ea6a029ea046ce8eae0d426bf8e485"
I20260927 03:28:27.464807 7453 raft_consensus.cc:3037] T 57792c0254fe43a0ab2b275272afaab2 P 38ea6a029ea046ce8eae0d426bf8e485 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:27.464802 7590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57792c0254fe43a0ab2b275272afaab2" candidate_uuid: "972f0abeb34849df82b4eca3335ec62d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc03843909b4012b156274ce3ca2535"
I20260927 03:28:27.464869 7590 raft_consensus.cc:3037] T 57792c0254fe43a0ab2b275272afaab2 P adc03843909b4012b156274ce3ca2535 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:27.466027 7590 raft_consensus.cc:2454] T 57792c0254fe43a0ab2b275272afaab2 P adc03843909b4012b156274ce3ca2535 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 972f0abeb34849df82b4eca3335ec62d in term 1.
I20260927 03:28:27.466058 7453 raft_consensus.cc:2454] T 57792c0254fe43a0ab2b275272afaab2 P 38ea6a029ea046ce8eae0d426bf8e485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 972f0abeb34849df82b4eca3335ec62d in term 1.
I20260927 03:28:27.466176 7659 leader_election.cc:302] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 972f0abeb34849df82b4eca3335ec62d, adc03843909b4012b156274ce3ca2535; no voters:
I20260927 03:28:27.466313 7803 raft_consensus.cc:2781] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:27.466498 7803 raft_consensus.cc:686] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d [term 1 LEADER]: Becoming Leader. State: Replica: 972f0abeb34849df82b4eca3335ec62d, State: Running, Role: LEADER
I20260927 03:28:27.466598 7803 consensus_queue.cc:237] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea6a029ea046ce8eae0d426bf8e485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39213 } } peers { permanent_uuid: "adc03843909b4012b156274ce3ca2535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "972f0abeb34849df82b4eca3335ec62d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34339 } }
I20260927 03:28:27.467412 6533 catalog_manager.cc:5495] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d reported cstate change: term changed from 0 to 1, leader changed from <none> to 972f0abeb34849df82b4eca3335ec62d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "972f0abeb34849df82b4eca3335ec62d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea6a029ea046ce8eae0d426bf8e485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adc03843909b4012b156274ce3ca2535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "972f0abeb34849df82b4eca3335ec62d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34339 } health_report { overall_health: HEALTHY } } }
I20260927 03:28:27.478219 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.475878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60896 (local address 127.0.98.3:34339)
0927 03:28:27.475919 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:27.475921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:27.475932 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:28:27.475969 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:27.475969 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:27.475989 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:27.476058 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:28:27.476181 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.476655 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.477427 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.477648 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.477724 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.477889 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:28:27.477992 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:28:27.478082 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:27.478120 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 03:28:27.478139 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:28:27.479899 7453 raft_consensus.cc:1261] T 57792c0254fe43a0ab2b275272afaab2 P 38ea6a029ea046ce8eae0d426bf8e485 [term 1 FOLLOWER]: Refusing update from remote peer 972f0abeb34849df82b4eca3335ec62d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:27.479991 7590 raft_consensus.cc:1261] T 57792c0254fe43a0ab2b275272afaab2 P adc03843909b4012b156274ce3ca2535 [term 1 FOLLOWER]: Refusing update from remote peer 972f0abeb34849df82b4eca3335ec62d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:27.480280 7803 consensus_queue.cc:1035] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38ea6a029ea046ce8eae0d426bf8e485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:27.480448 7804 consensus_queue.cc:1035] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc03843909b4012b156274ce3ca2535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:27.484352 7809 mvcc.cc:205] Tried to move back new op lower bound from 7333804881835261952 to 7333804881783836672. Current Snapshot: MvccSnapshot[applied={T|T < 7333804881835261952}]
I20260927 03:28:27.486140 6533 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:28:27.486240 6533 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:28:27.486656 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.482972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40375 (local address 127.0.0.1:57394)
0927 03:28:27.483117 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:27.483129 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:27.483180 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:27.483406 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:27.483407 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:27.483409 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:27.483477 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:27.483479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.484898 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.484899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:27.485697 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:27.485699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.486405 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.486406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:27.486477 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.486480 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:28:27.486550 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 03:28:27.486593 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260927 03:28:27.486655 7505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.483032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57394 (local address 127.0.98.60:40375)
0927 03:28:27.483099 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:27.483102 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:27.483166 (+ 64us) server_negotiation.cc:407] Connection header received
0927 03:28:27.483295 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:27.483296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:27.483320 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:27.483394 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:27.483559 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.484250 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.485795 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.486211 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.486302 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.486373 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:27.486555 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 03:28:27.486590 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:28:27.487030 7325 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.10] Dedup: 1.7->[1.8-1.10]
I20260927 03:28:27.487910 7359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abc21e02fdc0470ebb9a8fce09d9f081. Latest consensus state: current_term: 1 leader_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: false } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: true } } }
I20260927 03:28:27.488006 7359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896 [sys.catalog]: This master's current role is: LEARNER
I20260927 03:28:27.488090 7808 mvcc.cc:205] Tried to move back new op lower bound from 7333804881835261952 to 7333804881783836672. Current Snapshot: MvccSnapshot[applied={T|T < 7333804881835261952}]
I20260927 03:28:27.488349 6533 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081: Sending DeleteTablet for 3 replicas of tablet 57792c0254fe43a0ab2b275272afaab2
I20260927 03:28:27.488647 7570 tablet_service.cc:1514] Processing DeleteTablet for tablet 57792c0254fe43a0ab2b275272afaab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59478
I20260927 03:28:27.488649 7569 tablet_service.cc:1514] Processing DeleteTablet for tablet 57792c0254fe43a0ab2b275272afaab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59478
I20260927 03:28:27.488957 7093 sys_catalog.cc:455] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: false } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: true } } }
I20260927 03:28:27.489049 7093 sys_catalog.cc:458] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:27.489216 7705 tablet_service.cc:1514] Processing DeleteTablet for tablet 57792c0254fe43a0ab2b275272afaab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60892
I20260927 03:28:27.489408 7094 raft_consensus.cc:1050] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081: attempting to promote NON_VOTER 510a4c04062b4014b9466b0a818c5896 to VOTER
I20260927 03:28:27.489393 7433 tablet_service.cc:1514] Processing DeleteTablet for tablet 57792c0254fe43a0ab2b275272afaab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54694
I20260927 03:28:27.489216 7706 tablet_service.cc:1514] Processing DeleteTablet for tablet 57792c0254fe43a0ab2b275272afaab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60892
I20260927 03:28:27.489528 7094 consensus_queue.cc:237] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: false } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: false } }
I20260927 03:28:27.489923 7325 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896 [term 1 LEARNER]: Refusing update from remote peer abc21e02fdc0470ebb9a8fce09d9f081: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20260927 03:28:27.490139 7280 consensus_queue.cc:1035] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260927 03:28:27.490199 7142 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [term 1 FOLLOWER]: Refusing update from remote peer abc21e02fdc0470ebb9a8fce09d9f081: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20260927 03:28:27.490517 7280 consensus_queue.cc:1035] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260927 03:28:27.491065 7093 raft_consensus.cc:2932] T 00000000000000000000000000000000 P abc21e02fdc0470ebb9a8fce09d9f081 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 7 to 12, 510a4c04062b4014b9466b0a818c5896 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: false } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: false } } }
I20260927 03:28:27.491492 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 6501
I20260927 03:28:27.491585 7142 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 7 to 12, 510a4c04062b4014b9466b0a818c5896 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: false } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: false } } }
I20260927 03:28:27.492620 7818 tablet_replica.cc:330] T 57792c0254fe43a0ab2b275272afaab2 P adc03843909b4012b156274ce3ca2535: stopping tablet replica
I20260927 03:28:27.492748 7818 raft_consensus.cc:2229] T 57792c0254fe43a0ab2b275272afaab2 P adc03843909b4012b156274ce3ca2535 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:28:27.492820 7818 raft_consensus.cc:2258] T 57792c0254fe43a0ab2b275272afaab2 P adc03843909b4012b156274ce3ca2535 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:27.492972 7821 tablet_replica.cc:330] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d: stopping tablet replica
I20260927 03:28:27.493067 7821 raft_consensus.cc:2229] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:28:27.493163 7821 raft_consensus.cc:2258] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:27.493654 7821 ts_tablet_manager.cc:1905] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:27.495303 7821 ts_tablet_manager.cc:1918] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:28:27.495388 7821 log.cc:1196] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/wal/wals/57792c0254fe43a0ab2b275272afaab2
I20260927 03:28:27.495718 7821 ts_tablet_manager.cc:1939] T 57792c0254fe43a0ab2b275272afaab2 P 972f0abeb34849df82b4eca3335ec62d: Deleting consensus metadata
W20260927 03:28:27.498386 7305 connection.cc:535] server connection from 127.0.0.1:57394 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 03:28:27.498404 7659 connection.cc:535] server connection from 127.0.0.1:60892 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 03:28:27.499042 7117 connection.cc:535] server connection from 127.0.0.1:48442 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 03:28:27.499492 7823 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: false } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: false } } }
I20260927 03:28:27.499709 7823 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:28:27.499866 7820 tablet_replica.cc:330] T 57792c0254fe43a0ab2b275272afaab2 P 38ea6a029ea046ce8eae0d426bf8e485: stopping tablet replica
I20260927 03:28:27.499974 7820 raft_consensus.cc:2229] T 57792c0254fe43a0ab2b275272afaab2 P 38ea6a029ea046ce8eae0d426bf8e485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:28:27.500001 7505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.490532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57396 (local address 127.0.98.60:40375)
0927 03:28:27.490567 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:27.490568 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:27.490576 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:28:27.490595 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:27.490596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:27.490607 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:27.490673 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:27.492024 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.492698 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.494133 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.494479 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.494522 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.496486 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:27.496488 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:27.496488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:27.496495 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:27.498421 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:27.498881 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:27.498882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:27.498882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:27.498947 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:27.499186 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:27.499187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:27.499188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:27.499321 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:27.499396 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:27.499792 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 03:28:27.499878 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:28:27.500097 7820 raft_consensus.cc:2258] T 57792c0254fe43a0ab2b275272afaab2 P 38ea6a029ea046ce8eae0d426bf8e485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:27.501238 7820 ts_tablet_manager.cc:1905] T 57792c0254fe43a0ab2b275272afaab2 P 38ea6a029ea046ce8eae0d426bf8e485: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:27.502893 7820 ts_tablet_manager.cc:1918] T 57792c0254fe43a0ab2b275272afaab2 P 38ea6a029ea046ce8eae0d426bf8e485: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:28:27.502979 7820 log.cc:1196] T 57792c0254fe43a0ab2b275272afaab2 P 38ea6a029ea046ce8eae0d426bf8e485: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-0/wal/wals/57792c0254fe43a0ab2b275272afaab2
I20260927 03:28:27.503271 7820 ts_tablet_manager.cc:1939] T 57792c0254fe43a0ab2b275272afaab2 P 38ea6a029ea046ce8eae0d426bf8e485: Deleting consensus metadata
W20260927 03:28:27.503643 7387 connection.cc:441] server connection from 127.0.0.1:54694 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54694 (request call id 1) could send its response
I20260927 03:28:27.504559 7818 ts_tablet_manager.cc:1905] T 57792c0254fe43a0ab2b275272afaab2 P adc03843909b4012b156274ce3ca2535: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:27.506112 7818 ts_tablet_manager.cc:1918] T 57792c0254fe43a0ab2b275272afaab2 P adc03843909b4012b156274ce3ca2535: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:28:27.506193 7818 log.cc:1196] T 57792c0254fe43a0ab2b275272afaab2 P adc03843909b4012b156274ce3ca2535: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/wal/wals/57792c0254fe43a0ab2b275272afaab2
I20260927 03:28:27.506438 7818 ts_tablet_manager.cc:1939] T 57792c0254fe43a0ab2b275272afaab2 P adc03843909b4012b156274ce3ca2535: Deleting consensus metadata
W20260927 03:28:27.506975 7525 connection.cc:441] server connection from 127.0.0.1:59478 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59478 (request call id 1) could send its response
W20260927 03:28:27.507048 7525 connection.cc:441] server connection from 127.0.0.1:59478 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59478 (request call id 2) could send its response
I20260927 03:28:27.509467 7505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.501018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57402 (local address 127.0.98.60:40375)
0927 03:28:27.501052 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:27.501054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:27.501201 (+ 147us) server_negotiation.cc:407] Connection header received
0927 03:28:27.501295 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:27.501295 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:27.501306 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:27.501334 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:27.502176 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.502919 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.504186 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.504515 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.504561 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.506146 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:27.506147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:27.506148 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:27.506154 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:27.508243 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:27.508664 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:27.508665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:27.508665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:27.508722 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:27.508932 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:27.508933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:27.508934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:27.509054 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:27.509163 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:27.509278 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 03:28:27.509351 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:28:27.511233 7183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:27.501325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48470 (local address 127.0.98.61:34729)
0927 03:28:27.501409 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:27.501411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:27.503853 (+ 2442us) server_negotiation.cc:407] Connection header received
0927 03:28:27.503958 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:27.503959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:27.503972 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:27.504038 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:27.504735 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.505489 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.506528 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:27.506834 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:27.506880 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:27.508387 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:27.508388 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:27.508389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:27.508396 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:27.510066 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:27.510484 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:27.510485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:27.510485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:27.510540 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:27.510752 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:27.510753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:27.510754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:27.510863 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:27.510963 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:27.511067 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 03:28:27.511138 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 03:28:28.485500 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:28.485161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.98.1:46121)
0927 03:28:28.485349 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:28.485428 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":85,"threads_started":1}
W20260927 03:28:28.485817 7500 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41079 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:41079: connect: Connection refused (error 111)
I20260927 03:28:28.489073 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:28.488794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.98.3:50221)
0927 03:28:28.488971 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:28.489034 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":68,"threads_started":1}
W20260927 03:28:28.489254 7773 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41079 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:41079: connect: Connection refused (error 111)
I20260927 03:28:28.489621 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:28.489417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.98.2:33991)
0927 03:28:28.489516 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:28.489572 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":50,"threads_started":1}
W20260927 03:28:28.489813 7637 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41079 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:41079: connect: Connection refused (error 111)
I20260927 03:28:28.992103 7835 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader abc21e02fdc0470ebb9a8fce09d9f081)
I20260927 03:28:28.992262 7835 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: false } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: false } }
I20260927 03:28:28.992548 7835 leader_election.cc:288] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers abc21e02fdc0470ebb9a8fce09d9f081 (127.0.98.62:41079), 510a4c04062b4014b9466b0a818c5896 (127.0.98.60:40375)
I20260927 03:28:28.992810 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:28.992598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.0.1:46126)
0927 03:28:28.992710 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:28.992756 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":61,"threads_started":1}
W20260927 03:28:28.992969 7117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41079: connect: Connection refused (error 111)
I20260927 03:28:28.993484 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:28.993334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.0.1:46128)
0927 03:28:28.993402 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:28.993445 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 03:28:28.993592 7117 leader_election.cc:334] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer abc21e02fdc0470ebb9a8fce09d9f081 (127.0.98.62:41079): Network error: Client connection negotiation failed: client connection to 127.0.98.62:41079: connect: Connection refused (error 111)
I20260927 03:28:28.995771 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:28.992789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57408 (local address 127.0.98.60:40375)
0927 03:28:28.993027 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:28.993029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:28.993039 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:28:28.993225 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:28.993226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:28.993237 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:28.993307 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:28.993476 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:28.994137 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:28.995100 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:28.995523 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:28.995579 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:28.995589 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:28.995698 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 03:28:28.995723 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":99,"threads_started":1}
I20260927 03:28:28.995803 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:28.992677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40375 (local address 127.0.0.1:57408)
0927 03:28:28.992851 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:28.992861 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:28.993146 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:28.993330 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:28.993331 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:28.993334 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:28.993401 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:28.993403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:28.994216 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:28.994217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:28.995028 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:28.995031 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:28.995589 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:28.995590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:28.995639 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:28.995643 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:28:28.995692 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 03:28:28.995757 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":21,"thread_start_us":106,"threads_started":1}
I20260927 03:28:28.996075 7325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ab1b1713c6064c04a39dd6f981745533" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "510a4c04062b4014b9466b0a818c5896" is_pre_election: true
I20260927 03:28:28.996239 7325 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1b1713c6064c04a39dd6f981745533 in term 1.
I20260927 03:28:28.996430 7116 leader_election.cc:302] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 510a4c04062b4014b9466b0a818c5896, ab1b1713c6064c04a39dd6f981745533; no voters: abc21e02fdc0470ebb9a8fce09d9f081
I20260927 03:28:28.996536 7835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 03:28:28.996582 7835 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [term 1 FOLLOWER]: Starting leader election (detected failure of leader abc21e02fdc0470ebb9a8fce09d9f081)
I20260927 03:28:28.996611 7835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [term 1 FOLLOWER]: Advancing to term 2
I20260927 03:28:28.997328 7835 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: false } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: false } }
I20260927 03:28:28.997473 7835 leader_election.cc:288] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [CANDIDATE]: Term 2 election: Requested vote from peers abc21e02fdc0470ebb9a8fce09d9f081 (127.0.98.62:41079), 510a4c04062b4014b9466b0a818c5896 (127.0.98.60:40375)
I20260927 03:28:28.997607 7325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ab1b1713c6064c04a39dd6f981745533" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "510a4c04062b4014b9466b0a818c5896"
I20260927 03:28:28.997648 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:28.997565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.0.1:46132)
0927 03:28:28.997598 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:28.997626 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260927 03:28:28.997680 7325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896 [term 1 FOLLOWER]: Advancing to term 2
I20260927 03:28:28.997951 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:28.997873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.0.1:46140)
0927 03:28:28.997916 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:28.997938 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 03:28:28.998054 7117 leader_election.cc:334] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer abc21e02fdc0470ebb9a8fce09d9f081 (127.0.98.62:41079): Network error: Client connection negotiation failed: client connection to 127.0.98.62:41079: connect: Connection refused (error 111)
I20260927 03:28:28.998492 7325 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1b1713c6064c04a39dd6f981745533 in term 2.
I20260927 03:28:28.998648 7116 leader_election.cc:302] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 510a4c04062b4014b9466b0a818c5896, ab1b1713c6064c04a39dd6f981745533; no voters: abc21e02fdc0470ebb9a8fce09d9f081
I20260927 03:28:28.998747 7835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 03:28:28.998867 7835 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [term 2 LEADER]: Becoming Leader. State: Replica: ab1b1713c6064c04a39dd6f981745533, State: Running, Role: LEADER
I20260927 03:28:28.998931 7835 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: false } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: false } }
I20260927 03:28:28.999320 7840 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab1b1713c6064c04a39dd6f981745533. Latest consensus state: current_term: 2 leader_uuid: "ab1b1713c6064c04a39dd6f981745533" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: false } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: false } } }
I20260927 03:28:28.999493 7840 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:28.999711 7842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:28:29.000138 7842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:28:29.000196 7842 catalog_manager.cc:1259] Loaded cluster ID: 0d21507db5a54eaaba302abd4d3f957a
I20260927 03:28:29.000248 7842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:28:29.000422 7842 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:28:29.000509 7842 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533: Loaded TSK: 0
I20260927 03:28:29.000667 7842 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:28:29.055823 7132 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48470:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:28:29.057703 7132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:28:29.058833 7325 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896 [term 2 FOLLOWER]: Refusing update from remote peer ab1b1713c6064c04a39dd6f981745533: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 03:28:29.058914 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:29.058759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.0.1:46170)
0927 03:28:29.058821 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:29.058860 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 03:28:29.059175 7840 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [LEADER]: Connected to new peer: Peer: permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260927 03:28:29.059355 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:29.059229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.0.1:46172)
0927 03:28:29.059278 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:29.059304 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 03:28:29.059484 7117 consensus_peers.cc:487] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 -> Peer abc21e02fdc0470ebb9a8fce09d9f081 (127.0.98.62:41079): Couldn't send request to peer abc21e02fdc0470ebb9a8fce09d9f081. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 03:28:29.059703 7325 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896 [term 2 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 7 to 12, 510a4c04062b4014b9466b0a818c5896 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: false } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: false } } }
I20260927 03:28:29.060938 7846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ab1b1713c6064c04a39dd6f981745533" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: false } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: false } } }
I20260927 03:28:29.061053 7846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:28:29.061025 7844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab1b1713c6064c04a39dd6f981745533. Latest consensus state: current_term: 2 leader_uuid: "ab1b1713c6064c04a39dd6f981745533" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: false } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: false } } }
I20260927 03:28:29.061098 7844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:28:29.061396 7835 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ab1b1713c6064c04a39dd6f981745533" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: false } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: false } } }
I20260927 03:28:29.061491 7835 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533 [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:29.061596 7844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ab1b1713c6064c04a39dd6f981745533" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "abc21e02fdc0470ebb9a8fce09d9f081" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41079 } } peers { permanent_uuid: "ab1b1713c6064c04a39dd6f981745533" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34729 } attrs { promote: false } } peers { permanent_uuid: "510a4c04062b4014b9466b0a818c5896" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40375 } attrs { promote: false } } }
I20260927 03:28:29.061686 7844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510a4c04062b4014b9466b0a818c5896 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:28:29.061900 7847 mvcc.cc:205] Tried to move back new op lower bound from 7333804888302735360 to 7333804888060301312. Current Snapshot: MvccSnapshot[applied={T|T < 7333804888302735360}]
I20260927 03:28:29.066537 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:29.063421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39213 (local address 127.0.0.1:54704)
0927 03:28:29.063489 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:29.063502 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:29.063551 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:29.063870 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:29.063871 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:29.063873 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:29.063920 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:29.063922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:29.064895 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:29.064896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:29.065728 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:29.065730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:29.066296 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:29.066297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:29.066355 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:29.066360 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:28:29.066412 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 03:28:29.066460 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":69}
I20260927 03:28:29.066552 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:29.063499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54704 (local address 127.0.98.1:39213)
0927 03:28:29.063740 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:29.063742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:29.063751 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:28:29.063786 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:29.063787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:29.063799 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:29.063859 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:29.063997 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:29.064804 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:29.065798 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:29.066214 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:29.066297 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:29.066310 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:29.066433 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 03:28:29.066471 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":60,"threads_started":1}
I20260927 03:28:29.066877 7433 tablet_service.cc:1467] Processing CreateTablet for tablet 6aeeb19839a34309b5227c1663da12c0 (DEFAULT_TABLE table=test-table [id=bf9093e5319a43229d7ee5ee79fbc20a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:29.067034 7433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aeeb19839a34309b5227c1663da12c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:29.067123 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:29.063554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34339 (local address 127.0.0.1:60900)
0927 03:28:29.063627 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:29.063636 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:29.063678 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:29.064077 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:29.064078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:29.064082 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:29.064134 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:29.064136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:29.064957 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:29.064958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:29.066128 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:29.066130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:29.066865 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:29.066866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:29.066935 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:29.066937 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:28:29.067011 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 03:28:29.067055 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":34}
I20260927 03:28:29.067361 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:29.063639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60900 (local address 127.0.98.3:34339)
0927 03:28:29.063937 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:29.063938 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:29.063946 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:28:29.063969 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:29.063970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:29.063980 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:29.064069 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:29.064211 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:29.064890 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:29.066227 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:29.066804 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:29.066867 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:29.066878 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:29.067221 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 03:28:29.067292 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":51,"threads_started":1}
I20260927 03:28:29.067592 7705 tablet_service.cc:1467] Processing CreateTablet for tablet 6aeeb19839a34309b5227c1663da12c0 (DEFAULT_TABLE table=test-table [id=bf9093e5319a43229d7ee5ee79fbc20a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:29.067744 7705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aeeb19839a34309b5227c1663da12c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:29.068521 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:29.063658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44295 (local address 127.0.0.1:59484)
0927 03:28:29.063801 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:29.063810 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:29.063859 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:29.064150 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:29.064151 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:29.064153 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:29.064187 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:29.064188 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:29.064873 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:29.064874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:29.066366 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:29.066368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:29.067827 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:29.067828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:29.067899 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:29.067904 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:28:29.067956 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 03:28:29.068424 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":91,"thread_start_us":93,"threads_started":1}
I20260927 03:28:29.068756 7855 tablet_bootstrap.cc:492] T 6aeeb19839a34309b5227c1663da12c0 P 38ea6a029ea046ce8eae0d426bf8e485: Bootstrap starting.
I20260927 03:28:29.068969 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:29.063556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59484 (local address 127.0.98.2:44295)
0927 03:28:29.063774 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:29.063777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:29.063996 (+ 219us) server_negotiation.cc:407] Connection header received
0927 03:28:29.064030 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:29.064032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:29.064043 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:29.064106 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:29.064257 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:29.064764 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:29.066463 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:29.067609 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:29.067761 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:29.068771 (+ 1010us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:29.068806 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 03:28:29.068889 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":60,"threads_started":1}
I20260927 03:28:29.069186 7569 tablet_service.cc:1467] Processing CreateTablet for tablet 6aeeb19839a34309b5227c1663da12c0 (DEFAULT_TABLE table=test-table [id=bf9093e5319a43229d7ee5ee79fbc20a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:29.069235 7856 tablet_bootstrap.cc:492] T 6aeeb19839a34309b5227c1663da12c0 P 972f0abeb34849df82b4eca3335ec62d: Bootstrap starting.
I20260927 03:28:29.069324 7855 tablet_bootstrap.cc:654] T 6aeeb19839a34309b5227c1663da12c0 P 38ea6a029ea046ce8eae0d426bf8e485: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:29.069325 7569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aeeb19839a34309b5227c1663da12c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:29.069777 7856 tablet_bootstrap.cc:654] T 6aeeb19839a34309b5227c1663da12c0 P 972f0abeb34849df82b4eca3335ec62d: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:29.069847 7855 tablet_bootstrap.cc:492] T 6aeeb19839a34309b5227c1663da12c0 P 38ea6a029ea046ce8eae0d426bf8e485: No bootstrap required, opened a new log
I20260927 03:28:29.069885 7855 ts_tablet_manager.cc:1397] T 6aeeb19839a34309b5227c1663da12c0 P 38ea6a029ea046ce8eae0d426bf8e485: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:29.069983 7855 raft_consensus.cc:348] T 6aeeb19839a34309b5227c1663da12c0 P 38ea6a029ea046ce8eae0d426bf8e485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea6a029ea046ce8eae0d426bf8e485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39213 } } peers { permanent_uuid: "adc03843909b4012b156274ce3ca2535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "972f0abeb34849df82b4eca3335ec62d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34339 } }
I20260927 03:28:29.070037 7855 raft_consensus.cc:374] T 6aeeb19839a34309b5227c1663da12c0 P 38ea6a029ea046ce8eae0d426bf8e485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:29.070051 7855 raft_consensus.cc:729] T 6aeeb19839a34309b5227c1663da12c0 P 38ea6a029ea046ce8eae0d426bf8e485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ea6a029ea046ce8eae0d426bf8e485, State: Initialized, Role: FOLLOWER
I20260927 03:28:29.070111 7855 consensus_queue.cc:260] T 6aeeb19839a34309b5227c1663da12c0 P 38ea6a029ea046ce8eae0d426bf8e485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea6a029ea046ce8eae0d426bf8e485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39213 } } peers { permanent_uuid: "adc03843909b4012b156274ce3ca2535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "972f0abeb34849df82b4eca3335ec62d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34339 } }
I20260927 03:28:29.070410 7855 ts_tablet_manager.cc:1428] T 6aeeb19839a34309b5227c1663da12c0 P 38ea6a029ea046ce8eae0d426bf8e485: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:29.070660 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:29.070410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.98.1:36035)
0927 03:28:29.070581 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:29.070619 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":74,"threads_started":1}
I20260927 03:28:29.071120 7501 heartbeater.cc:503] Master 127.0.98.61:34729 was elected leader, sending a full tablet report...
I20260927 03:28:29.071163 7861 tablet_bootstrap.cc:492] T 6aeeb19839a34309b5227c1663da12c0 P adc03843909b4012b156274ce3ca2535: Bootstrap starting.
I20260927 03:28:29.071305 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:29.071006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.98.1:42643)
0927 03:28:29.071257 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:29.071282 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":73,"threads_started":1}
I20260927 03:28:29.071640 7861 tablet_bootstrap.cc:654] T 6aeeb19839a34309b5227c1663da12c0 P adc03843909b4012b156274ce3ca2535: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:29.071787 7856 tablet_bootstrap.cc:492] T 6aeeb19839a34309b5227c1663da12c0 P 972f0abeb34849df82b4eca3335ec62d: No bootstrap required, opened a new log
I20260927 03:28:29.071848 7856 ts_tablet_manager.cc:1397] T 6aeeb19839a34309b5227c1663da12c0 P 972f0abeb34849df82b4eca3335ec62d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 03:28:29.071990 7856 raft_consensus.cc:348] T 6aeeb19839a34309b5227c1663da12c0 P 972f0abeb34849df82b4eca3335ec62d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea6a029ea046ce8eae0d426bf8e485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39213 } } peers { permanent_uuid: "adc03843909b4012b156274ce3ca2535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "972f0abeb34849df82b4eca3335ec62d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34339 } }
I20260927 03:28:29.072085 7856 raft_consensus.cc:374] T 6aeeb19839a34309b5227c1663da12c0 P 972f0abeb34849df82b4eca3335ec62d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:29.072113 7856 raft_consensus.cc:729] T 6aeeb19839a34309b5227c1663da12c0 P 972f0abeb34849df82b4eca3335ec62d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 972f0abeb34849df82b4eca3335ec62d, State: Initialized, Role: FOLLOWER
I20260927 03:28:29.072170 7856 consensus_queue.cc:260] T 6aeeb19839a34309b5227c1663da12c0 P 972f0abeb34849df82b4eca3335ec62d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea6a029ea046ce8eae0d426bf8e485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39213 } } peers { permanent_uuid: "adc03843909b4012b156274ce3ca2535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "972f0abeb34849df82b4eca3335ec62d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34339 } }
I20260927 03:28:29.072235 7861 tablet_bootstrap.cc:492] T 6aeeb19839a34309b5227c1663da12c0 P adc03843909b4012b156274ce3ca2535: No bootstrap required, opened a new log
I20260927 03:28:29.072312 7861 ts_tablet_manager.cc:1397] T 6aeeb19839a34309b5227c1663da12c0 P adc03843909b4012b156274ce3ca2535: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:29.072322 7856 ts_tablet_manager.cc:1428] T 6aeeb19839a34309b5227c1663da12c0 P 972f0abeb34849df82b4eca3335ec62d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:29.072467 7861 raft_consensus.cc:348] T 6aeeb19839a34309b5227c1663da12c0 P adc03843909b4012b156274ce3ca2535 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea6a029ea046ce8eae0d426bf8e485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39213 } } peers { permanent_uuid: "adc03843909b4012b156274ce3ca2535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "972f0abeb34849df82b4eca3335ec62d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34339 } }
I20260927 03:28:29.072556 7861 raft_consensus.cc:374] T 6aeeb19839a34309b5227c1663da12c0 P adc03843909b4012b156274ce3ca2535 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:29.072588 7861 raft_consensus.cc:729] T 6aeeb19839a34309b5227c1663da12c0 P adc03843909b4012b156274ce3ca2535 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adc03843909b4012b156274ce3ca2535, State: Initialized, Role: FOLLOWER
I20260927 03:28:29.072669 7861 consensus_queue.cc:260] T 6aeeb19839a34309b5227c1663da12c0 P adc03843909b4012b156274ce3ca2535 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea6a029ea046ce8eae0d426bf8e485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39213 } } peers { permanent_uuid: "adc03843909b4012b156274ce3ca2535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "972f0abeb34849df82b4eca3335ec62d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34339 } }
I20260927 03:28:29.072842 7861 ts_tablet_manager.cc:1428] T 6aeeb19839a34309b5227c1663da12c0 P adc03843909b4012b156274ce3ca2535: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:29.072957 7774 heartbeater.cc:503] Master 127.0.98.61:34729 was elected leader, sending a full tablet report...
I20260927 03:28:29.073285 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:29.072540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.98.3:37501)
0927 03:28:29.072955 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:29.073250 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":73,"threads_started":1}
I20260927 03:28:29.073539 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:29.073153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.98.2:51497)
0927 03:28:29.073448 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:29.073501 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":81,"threads_started":1}
I20260927 03:28:29.073586 7638 heartbeater.cc:503] Master 127.0.98.61:34729 was elected leader, sending a full tablet report...
I20260927 03:28:29.073873 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:29.073713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.98.2:56993)
0927 03:28:29.073822 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:29.073847 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":74,"threads_started":1}
W20260927 03:28:29.130450 7502 tablet.cc:2369] T 6aeeb19839a34309b5227c1663da12c0 P 38ea6a029ea046ce8eae0d426bf8e485: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:28:29.156724 7775 tablet.cc:2369] T 6aeeb19839a34309b5227c1663da12c0 P 972f0abeb34849df82b4eca3335ec62d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:28:29.203176 7858 raft_consensus.cc:482] T 6aeeb19839a34309b5227c1663da12c0 P 38ea6a029ea046ce8eae0d426bf8e485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:28:29.203287 7858 raft_consensus.cc:504] T 6aeeb19839a34309b5227c1663da12c0 P 38ea6a029ea046ce8eae0d426bf8e485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea6a029ea046ce8eae0d426bf8e485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39213 } } peers { permanent_uuid: "adc03843909b4012b156274ce3ca2535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "972f0abeb34849df82b4eca3335ec62d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34339 } }
I20260927 03:28:29.203644 7858 leader_election.cc:288] T 6aeeb19839a34309b5227c1663da12c0 P 38ea6a029ea046ce8eae0d426bf8e485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adc03843909b4012b156274ce3ca2535 (127.0.98.2:44295), 972f0abeb34849df82b4eca3335ec62d (127.0.98.3:34339)
I20260927 03:28:29.207798 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:29.203905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53707 (local address 127.0.98.3:34339)
0927 03:28:29.203949 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:29.203951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:29.203962 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:28:29.204051 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:29.204052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:29.204064 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:29.204128 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:29.204358 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:29.205098 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:29.206831 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:29.207404 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:29.207498 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:29.207512 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:29.207675 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 03:28:29.207719 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:28:29.207813 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:29.203829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34339 (local address 127.0.98.1:53707)
0927 03:28:29.203897 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:29.203908 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:29.203973 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:29.204152 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:29.204153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:29.204155 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:29.204267 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:29.204269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:29.205216 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:29.205217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:29.206698 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:29.206701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:29.207525 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:29.207526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:29.207602 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:29.207604 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:28:29.207655 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 03:28:29.207715 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":112}
I20260927 03:28:29.207841 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:29.203876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49363 (local address 127.0.98.2:44295)
0927 03:28:29.203924 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:29.203927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:29.203962 (+ 35us) server_negotiation.cc:407] Connection header received
0927 03:28:29.204052 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:29.204052 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:29.204062 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:29.204128 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:29.204358 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:29.205125 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:29.206832 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:29.207391 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:29.207509 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:29.207521 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:29.207675 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 03:28:29.207714 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:28:29.207813 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:29.203815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44295 (local address 127.0.98.1:49363)
0927 03:28:29.203894 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:29.203907 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:29.203973 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:29.204152 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:29.204153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:29.204155 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:29.204267 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:29.204269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:29.205216 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:29.205217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:29.206698 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:29.206701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:29.207505 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:29.207507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:29.207586 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:29.207592 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 03:28:29.207655 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 03:28:29.207715 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":137}
I20260927 03:28:29.208103 7590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aeeb19839a34309b5227c1663da12c0" candidate_uuid: "38ea6a029ea046ce8eae0d426bf8e485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc03843909b4012b156274ce3ca2535" is_pre_election: true
I20260927 03:28:29.208105 7726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aeeb19839a34309b5227c1663da12c0" candidate_uuid: "38ea6a029ea046ce8eae0d426bf8e485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "972f0abeb34849df82b4eca3335ec62d" is_pre_election: true
I20260927 03:28:29.208200 7590 raft_consensus.cc:2454] T 6aeeb19839a34309b5227c1663da12c0 P adc03843909b4012b156274ce3ca2535 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38ea6a029ea046ce8eae0d426bf8e485 in term 0.
I20260927 03:28:29.208220 7726 raft_consensus.cc:2454] T 6aeeb19839a34309b5227c1663da12c0 P 972f0abeb34849df82b4eca3335ec62d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38ea6a029ea046ce8eae0d426bf8e485 in term 0.
I20260927 03:28:29.208369 7388 leader_election.cc:302] T 6aeeb19839a34309b5227c1663da12c0 P 38ea6a029ea046ce8eae0d426bf8e485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38ea6a029ea046ce8eae0d426bf8e485, 972f0abeb34849df82b4eca3335ec62d; no voters:
I20260927 03:28:29.208546 7858 raft_consensus.cc:2781] T 6aeeb19839a34309b5227c1663da12c0 P 38ea6a029ea046ce8eae0d426bf8e485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:28:29.208590 7858 raft_consensus.cc:482] T 6aeeb19839a34309b5227c1663da12c0 P 38ea6a029ea046ce8eae0d426bf8e485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:28:29.208608 7858 raft_consensus.cc:3037] T 6aeeb19839a34309b5227c1663da12c0 P 38ea6a029ea046ce8eae0d426bf8e485 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:29.209455 7858 raft_consensus.cc:504] T 6aeeb19839a34309b5227c1663da12c0 P 38ea6a029ea046ce8eae0d426bf8e485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea6a029ea046ce8eae0d426bf8e485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39213 } } peers { permanent_uuid: "adc03843909b4012b156274ce3ca2535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "972f0abeb34849df82b4eca3335ec62d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34339 } }
I20260927 03:28:29.209599 7858 leader_election.cc:288] T 6aeeb19839a34309b5227c1663da12c0 P 38ea6a029ea046ce8eae0d426bf8e485 [CANDIDATE]: Term 1 election: Requested vote from peers adc03843909b4012b156274ce3ca2535 (127.0.98.2:44295), 972f0abeb34849df82b4eca3335ec62d (127.0.98.3:34339)
I20260927 03:28:29.209784 7590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aeeb19839a34309b5227c1663da12c0" candidate_uuid: "38ea6a029ea046ce8eae0d426bf8e485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc03843909b4012b156274ce3ca2535"
I20260927 03:28:29.209784 7726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aeeb19839a34309b5227c1663da12c0" candidate_uuid: "38ea6a029ea046ce8eae0d426bf8e485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "972f0abeb34849df82b4eca3335ec62d"
I20260927 03:28:29.209856 7726 raft_consensus.cc:3037] T 6aeeb19839a34309b5227c1663da12c0 P 972f0abeb34849df82b4eca3335ec62d [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:29.209856 7590 raft_consensus.cc:3037] T 6aeeb19839a34309b5227c1663da12c0 P adc03843909b4012b156274ce3ca2535 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:29.210615 7590 raft_consensus.cc:2454] T 6aeeb19839a34309b5227c1663da12c0 P adc03843909b4012b156274ce3ca2535 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38ea6a029ea046ce8eae0d426bf8e485 in term 1.
I20260927 03:28:29.210675 7726 raft_consensus.cc:2454] T 6aeeb19839a34309b5227c1663da12c0 P 972f0abeb34849df82b4eca3335ec62d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38ea6a029ea046ce8eae0d426bf8e485 in term 1.
I20260927 03:28:29.210822 7386 leader_election.cc:302] T 6aeeb19839a34309b5227c1663da12c0 P 38ea6a029ea046ce8eae0d426bf8e485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38ea6a029ea046ce8eae0d426bf8e485, adc03843909b4012b156274ce3ca2535; no voters:
I20260927 03:28:29.210924 7858 raft_consensus.cc:2781] T 6aeeb19839a34309b5227c1663da12c0 P 38ea6a029ea046ce8eae0d426bf8e485 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:29.211100 7858 raft_consensus.cc:686] T 6aeeb19839a34309b5227c1663da12c0 P 38ea6a029ea046ce8eae0d426bf8e485 [term 1 LEADER]: Becoming Leader. State: Replica: 38ea6a029ea046ce8eae0d426bf8e485, State: Running, Role: LEADER
I20260927 03:28:29.211186 7858 consensus_queue.cc:237] T 6aeeb19839a34309b5227c1663da12c0 P 38ea6a029ea046ce8eae0d426bf8e485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea6a029ea046ce8eae0d426bf8e485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39213 } } peers { permanent_uuid: "adc03843909b4012b156274ce3ca2535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } } peers { permanent_uuid: "972f0abeb34849df82b4eca3335ec62d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34339 } }
I20260927 03:28:29.211452 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:29.211306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.98.1:60863)
0927 03:28:29.211344 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:29.211383 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260927 03:28:29.212085 7131 catalog_manager.cc:5495] T 6aeeb19839a34309b5227c1663da12c0 P 38ea6a029ea046ce8eae0d426bf8e485 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38ea6a029ea046ce8eae0d426bf8e485 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "38ea6a029ea046ce8eae0d426bf8e485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea6a029ea046ce8eae0d426bf8e485" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adc03843909b4012b156274ce3ca2535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "972f0abeb34849df82b4eca3335ec62d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34339 } health_report { overall_health: UNKNOWN } } }
I20260927 03:28:29.249169 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:29.246643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54706 (local address 127.0.98.1:39213)
0927 03:28:29.246704 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:29.246705 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:29.246713 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:28:29.246781 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:29.246782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:29.246819 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:29.246907 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:28:29.247057 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:29.247608 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:29.248353 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:29.248560 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:29.248637 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:29.248814 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:28:29.248917 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:28:29.249012 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:29.249050 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 03:28:29.249072 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 03:28:29.250350 7590 raft_consensus.cc:1261] T 6aeeb19839a34309b5227c1663da12c0 P adc03843909b4012b156274ce3ca2535 [term 1 FOLLOWER]: Refusing update from remote peer 38ea6a029ea046ce8eae0d426bf8e485: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:29.250481 7726 raft_consensus.cc:1261] T 6aeeb19839a34309b5227c1663da12c0 P 972f0abeb34849df82b4eca3335ec62d [term 1 FOLLOWER]: Refusing update from remote peer 38ea6a029ea046ce8eae0d426bf8e485: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:29.250713 7869 consensus_queue.cc:1035] T 6aeeb19839a34309b5227c1663da12c0 P 38ea6a029ea046ce8eae0d426bf8e485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc03843909b4012b156274ce3ca2535" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:29.250840 7858 consensus_queue.cc:1035] T 6aeeb19839a34309b5227c1663da12c0 P 38ea6a029ea046ce8eae0d426bf8e485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "972f0abeb34849df82b4eca3335ec62d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:29.252028 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:29.251836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.98.2:33239)
0927 03:28:29.251963 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:29.251999 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 03:28:29.252131 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:29.251815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.98.3:50829)
0927 03:28:29.252062 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:29.252097 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218}
I20260927 03:28:29.252926 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:29.252836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.98.1:36319)
0927 03:28:29.252873 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:29.252899 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 03:28:29.253320 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:29.253191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.98.1:34573)
0927 03:28:29.253260 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:29.253285 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 03:28:29.253479 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:29.253358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.98.2:36189)
0927 03:28:29.253408 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:29.253438 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 03:28:29.254204 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:29.254067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41079 (local address 127.0.98.3:54753)
0927 03:28:29.254147 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:29.254173 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 03:28:29.254333 7874 mvcc.cc:205] Tried to move back new op lower bound from 7333804889087389696 to 7333804888929579008. Current Snapshot: MvccSnapshot[applied={T|T < 7333804889087389696}]
I20260927 03:28:29.255380 7132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:28:29.255496 7132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:28:29.256722 7132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab1b1713c6064c04a39dd6f981745533: Sending DeleteTablet for 3 replicas of tablet 6aeeb19839a34309b5227c1663da12c0
I20260927 03:28:29.257016 7569 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aeeb19839a34309b5227c1663da12c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:29 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:59484
I20260927 03:28:29.257058 7705 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aeeb19839a34309b5227c1663da12c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:29 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:60900
I20260927 03:28:29.257098 7433 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aeeb19839a34309b5227c1663da12c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:29 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:54704
I20260927 03:28:29.257249 7883 tablet_replica.cc:330] T 6aeeb19839a34309b5227c1663da12c0 P 38ea6a029ea046ce8eae0d426bf8e485: stopping tablet replica
I20260927 03:28:29.257310 7883 raft_consensus.cc:2229] T 6aeeb19839a34309b5227c1663da12c0 P 38ea6a029ea046ce8eae0d426bf8e485 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:28:29.257395 7883 raft_consensus.cc:2258] T 6aeeb19839a34309b5227c1663da12c0 P 38ea6a029ea046ce8eae0d426bf8e485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:29.257475 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 7282
I20260927 03:28:29.258016 7882 tablet_replica.cc:330] T 6aeeb19839a34309b5227c1663da12c0 P 972f0abeb34849df82b4eca3335ec62d: stopping tablet replica
I20260927 03:28:29.258069 7882 raft_consensus.cc:2229] T 6aeeb19839a34309b5227c1663da12c0 P 972f0abeb34849df82b4eca3335ec62d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:28:29.258105 7882 raft_consensus.cc:2258] T 6aeeb19839a34309b5227c1663da12c0 P 972f0abeb34849df82b4eca3335ec62d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:29.258190 7881 tablet_replica.cc:330] T 6aeeb19839a34309b5227c1663da12c0 P adc03843909b4012b156274ce3ca2535: stopping tablet replica
I20260927 03:28:29.258239 7881 raft_consensus.cc:2229] T 6aeeb19839a34309b5227c1663da12c0 P adc03843909b4012b156274ce3ca2535 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:28:29.258282 7881 raft_consensus.cc:2258] T 6aeeb19839a34309b5227c1663da12c0 P adc03843909b4012b156274ce3ca2535 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:29.258657 7881 ts_tablet_manager.cc:1905] T 6aeeb19839a34309b5227c1663da12c0 P adc03843909b4012b156274ce3ca2535: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:29.258827 7882 ts_tablet_manager.cc:1905] T 6aeeb19839a34309b5227c1663da12c0 P 972f0abeb34849df82b4eca3335ec62d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:29.260301 7881 ts_tablet_manager.cc:1918] T 6aeeb19839a34309b5227c1663da12c0 P adc03843909b4012b156274ce3ca2535: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:28:29.260356 7881 log.cc:1196] T 6aeeb19839a34309b5227c1663da12c0 P adc03843909b4012b156274ce3ca2535: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-1/wal/wals/6aeeb19839a34309b5227c1663da12c0
I20260927 03:28:29.260540 7881 ts_tablet_manager.cc:1939] T 6aeeb19839a34309b5227c1663da12c0 P adc03843909b4012b156274ce3ca2535: Deleting consensus metadata
I20260927 03:28:29.260864 7116 catalog_manager.cc:4841] TS adc03843909b4012b156274ce3ca2535 (127.0.98.2:44295): tablet 6aeeb19839a34309b5227c1663da12c0 (table test-table [id=bf9093e5319a43229d7ee5ee79fbc20a]) successfully deleted
I20260927 03:28:29.260900 7882 ts_tablet_manager.cc:1918] T 6aeeb19839a34309b5227c1663da12c0 P 972f0abeb34849df82b4eca3335ec62d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:28:29.261030 7882 log.cc:1196] T 6aeeb19839a34309b5227c1663da12c0 P 972f0abeb34849df82b4eca3335ec62d: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790479675865103-392-0/minicluster-data/ts-2/wal/wals/6aeeb19839a34309b5227c1663da12c0
I20260927 03:28:29.261272 7882 ts_tablet_manager.cc:1939] T 6aeeb19839a34309b5227c1663da12c0 P 972f0abeb34849df82b4eca3335ec62d: Deleting consensus metadata
I20260927 03:28:29.261749 7118 catalog_manager.cc:4841] TS 972f0abeb34849df82b4eca3335ec62d (127.0.98.3:34339): tablet 6aeeb19839a34309b5227c1663da12c0 (table test-table [id=bf9093e5319a43229d7ee5ee79fbc20a]) successfully deleted
I20260927 03:28:29.264287 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:29.264173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39213 (local address 127.0.0.1:54710)
0927 03:28:29.264225 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:29.264254 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 03:28:29.264333 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 7507
W20260927 03:28:29.264492 7118 catalog_manager.cc:4584] TS 38ea6a029ea046ce8eae0d426bf8e485 (127.0.98.1:39213): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6aeeb19839a34309b5227c1663da12c0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39213: connect: Connection refused (error 111)
I20260927 03:28:29.270313 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 7643
I20260927 03:28:29.275998 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 6998
I20260927 03:28:29.281118 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 7186
2026-09-27T03:28:29Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5671 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:28:29 dist-test-slave-j0hf krb5kdc[7886](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:28:29 dist-test-slave-j0hf krb5kdc[7886](info): set up 2 sockets
Sep 27 03:28:29 dist-test-slave-j0hf krb5kdc[7886](info): commencing operation
krb5kdc: starting...
W20260927 03:28:30.323939 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:28:30 dist-test-slave-j0hf krb5kdc[7886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:28:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:28:30Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 03:28:30.456202 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34037
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:30.537312 7899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:30.537550 7899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:30.537587 7899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:30.538810 7899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:28:30.538875 7899 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 03:28:30.538902 7899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:30.538935 7899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:30.538968 7899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:28:30.539003 7899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:28:30.540547 7899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34037
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:30.540844 7899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:30.541071 7899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:30.546072 7907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:30.546082 7905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:30.546149 7904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:30.546272 7899 server_base.cc:1034] running on GCE node
I20260927 03:28:30.546566 7899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:30.546875 7899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:30.548035 7899 hybrid_clock.cc:648] HybridClock initialized: now 1790479710548016 us; error 54 us; skew 500 ppm
Sep 27 03:28:30 dist-test-slave-j0hf krb5kdc[7886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479710, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:30.550395 7899 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 03:28:30.550891 7899 webserver.cc:460] Webserver started at http://127.0.98.62:42537/ using document root <none> and password file <none>
I20260927 03:28:30.551115 7899 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:30.551182 7899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:30.551334 7899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:30.552439 7899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "456730cb44ed4c2f8cb251842ef9f9bb"
format_stamp: "Formatted at 2026-09-27 03:28:30 on dist-test-slave-j0hf"
server_key: "130b5a94c7c8c1586a93b6858c270c09"
server_key_iv: "aaee037804431ee840257286ea00de77"
server_key_version: "encryptionkey@0"
I20260927 03:28:30.552809 7899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "456730cb44ed4c2f8cb251842ef9f9bb"
format_stamp: "Formatted at 2026-09-27 03:28:30 on dist-test-slave-j0hf"
server_key: "130b5a94c7c8c1586a93b6858c270c09"
server_key_iv: "aaee037804431ee840257286ea00de77"
server_key_version: "encryptionkey@0"
I20260927 03:28:30.554097 7899 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:28:30.554785 7914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:30.554957 7899 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:30.555053 7899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "456730cb44ed4c2f8cb251842ef9f9bb"
format_stamp: "Formatted at 2026-09-27 03:28:30 on dist-test-slave-j0hf"
server_key: "130b5a94c7c8c1586a93b6858c270c09"
server_key_iv: "aaee037804431ee840257286ea00de77"
server_key_version: "encryptionkey@0"
I20260927 03:28:30.555137 7899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:30.563024 7899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:30.564851 7899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:30.565004 7899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:30.569746 7899 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34037
I20260927 03:28:30.569787 7976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34037 every 8 connection(s)
I20260927 03:28:30.570122 7899 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:28:30.570643 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 7899
I20260927 03:28:30.570789 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:28:30.570760 7977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:30.571035 392 external_mini_cluster.cc:1428] Setting key 392170beede2eb7240b99cafa60d2623
I20260927 03:28:30.574087 7977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 456730cb44ed4c2f8cb251842ef9f9bb: Bootstrap starting.
I20260927 03:28:30.574755 7977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 456730cb44ed4c2f8cb251842ef9f9bb: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:30.575065 7977 log.cc:824] T 00000000000000000000000000000000 P 456730cb44ed4c2f8cb251842ef9f9bb: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:30.575737 7977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 456730cb44ed4c2f8cb251842ef9f9bb: No bootstrap required, opened a new log
I20260927 03:28:30.577669 7977 raft_consensus.cc:348] T 00000000000000000000000000000000 P 456730cb44ed4c2f8cb251842ef9f9bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456730cb44ed4c2f8cb251842ef9f9bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } }
I20260927 03:28:30.577826 7977 raft_consensus.cc:374] T 00000000000000000000000000000000 P 456730cb44ed4c2f8cb251842ef9f9bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:30.577875 7977 raft_consensus.cc:729] T 00000000000000000000000000000000 P 456730cb44ed4c2f8cb251842ef9f9bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 456730cb44ed4c2f8cb251842ef9f9bb, State: Initialized, Role: FOLLOWER
I20260927 03:28:30.578047 7977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 456730cb44ed4c2f8cb251842ef9f9bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456730cb44ed4c2f8cb251842ef9f9bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } }
I20260927 03:28:30.578145 7977 raft_consensus.cc:388] T 00000000000000000000000000000000 P 456730cb44ed4c2f8cb251842ef9f9bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:28:30.578184 7977 raft_consensus.cc:482] T 00000000000000000000000000000000 P 456730cb44ed4c2f8cb251842ef9f9bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:28:30.578233 7977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 456730cb44ed4c2f8cb251842ef9f9bb [term 0 FOLLOWER]: Advancing to term 1
Sep 27 03:28:30 dist-test-slave-j0hf krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 03:28:30.578763 7977 raft_consensus.cc:504] T 00000000000000000000000000000000 P 456730cb44ed4c2f8cb251842ef9f9bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456730cb44ed4c2f8cb251842ef9f9bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } }
I20260927 03:28:30.578882 7977 leader_election.cc:302] T 00000000000000000000000000000000 P 456730cb44ed4c2f8cb251842ef9f9bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 456730cb44ed4c2f8cb251842ef9f9bb; no voters:
I20260927 03:28:30.579071 7977 leader_election.cc:288] T 00000000000000000000000000000000 P 456730cb44ed4c2f8cb251842ef9f9bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:28:30.579118 7982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 456730cb44ed4c2f8cb251842ef9f9bb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:30.579308 7982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 456730cb44ed4c2f8cb251842ef9f9bb [term 1 LEADER]: Becoming Leader. State: Replica: 456730cb44ed4c2f8cb251842ef9f9bb, State: Running, Role: LEADER
I20260927 03:28:30.579340 7977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 456730cb44ed4c2f8cb251842ef9f9bb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:28:30.579406 7982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 456730cb44ed4c2f8cb251842ef9f9bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456730cb44ed4c2f8cb251842ef9f9bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } }
I20260927 03:28:30.579790 7982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 456730cb44ed4c2f8cb251842ef9f9bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 456730cb44ed4c2f8cb251842ef9f9bb. Latest consensus state: current_term: 1 leader_uuid: "456730cb44ed4c2f8cb251842ef9f9bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456730cb44ed4c2f8cb251842ef9f9bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } } }
I20260927 03:28:30.579878 7982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 456730cb44ed4c2f8cb251842ef9f9bb [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:30.579993 7984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 456730cb44ed4c2f8cb251842ef9f9bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "456730cb44ed4c2f8cb251842ef9f9bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456730cb44ed4c2f8cb251842ef9f9bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34037 } } }
I20260927 03:28:30.580060 7984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 456730cb44ed4c2f8cb251842ef9f9bb [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:30.580094 7989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:28:30.580631 7989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:28:30.582476 7989 catalog_manager.cc:1347] Generated new cluster ID: 94bc2e669d434ef58962f4b00ac23e2c
I20260927 03:28:30.582520 7989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:28:30.583221 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:30.571654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40412 (local address 127.0.98.62:34037)
0927 03:28:30.571848 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:30.571858 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:30.571891 (+ 33us) server_negotiation.cc:407] Connection header received
0927 03:28:30.572384 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:30.572393 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:30.572466 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:30.572621 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:30.573279 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:30.574650 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:30.575530 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:30.576015 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:30.576156 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:30.578851 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:30.578855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:30.578856 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:30.578869 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:30.581228 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:30.581670 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:30.581673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:30.581674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:30.581769 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:30.581977 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:30.581978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:30.581979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:30.582289 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:30.582430 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:30.582897 (+ 467us) server_negotiation.cc:299] Negotiation successful
0927 03:28:30.583035 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":86,"threads_started":1}
I20260927 03:28:30.594416 7989 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:28:30.594969 7989 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:28:30.601197 7989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 456730cb44ed4c2f8cb251842ef9f9bb: Generated new TSK 0
I20260927 03:28:30.601383 7989 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 03:28:30.630635 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34037
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:30.721094 8003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:30.721349 8003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:30.721392 8003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:30.722623 8003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:30.722697 8003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:30.722733 8003 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:28:30.722775 8003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:28:30.724298 8003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:30.724579 8003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:30.724817 8003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:30.730038 8008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:30.730077 8009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:30.730193 8003 server_base.cc:1034] running on GCE node
W20260927 03:28:30.730046 8011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:30.730492 8003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:30.730772 8003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:30.731946 8003 hybrid_clock.cc:648] HybridClock initialized: now 1790479710731928 us; error 29 us; skew 500 ppm
Sep 27 03:28:30 dist-test-slave-j0hf krb5kdc[7886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479710, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:30.734618 8003 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 03:28:30.735144 8003 webserver.cc:460] Webserver started at http://127.0.98.1:39389/ using document root <none> and password file <none>
I20260927 03:28:30.735388 8003 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:30.735487 8003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:30.735647 8003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:30.736737 8003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-0/data/instance:
uuid: "7c403450386c4d7290117df544ead104"
format_stamp: "Formatted at 2026-09-27 03:28:30 on dist-test-slave-j0hf"
server_key: "75bf03911eb15bcbcc9871211998cfe1"
server_key_iv: "493174bf6c390f11c62a79256d812e63"
server_key_version: "encryptionkey@0"
I20260927 03:28:30.737106 8003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7c403450386c4d7290117df544ead104"
format_stamp: "Formatted at 2026-09-27 03:28:30 on dist-test-slave-j0hf"
server_key: "75bf03911eb15bcbcc9871211998cfe1"
server_key_iv: "493174bf6c390f11c62a79256d812e63"
server_key_version: "encryptionkey@0"
I20260927 03:28:30.738461 8003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:28:30.739205 8018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:30.739373 8003 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:28:30.739492 8003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "7c403450386c4d7290117df544ead104"
format_stamp: "Formatted at 2026-09-27 03:28:30 on dist-test-slave-j0hf"
server_key: "75bf03911eb15bcbcc9871211998cfe1"
server_key_iv: "493174bf6c390f11c62a79256d812e63"
server_key_version: "encryptionkey@0"
I20260927 03:28:30.739578 8003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:30.762454 8003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:30.764940 8003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:30.765103 8003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:30.765787 8003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:30.765872 8003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:30.765911 8003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:30.765956 8003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:30.772924 8003 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45955
Sep 27 03:28:30 dist-test-slave-j0hf krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479710, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 03:28:30.773037 8133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45955 every 8 connection(s)
I20260927 03:28:30.773311 8003 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:28:30.776173 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 8003
I20260927 03:28:30.776293 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:28:30.776400 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:30.766280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34019 (local address 127.0.98.62:34037)
0927 03:28:30.766336 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:30.766339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:30.766589 (+ 250us) server_negotiation.cc:407] Connection header received
0927 03:28:30.767076 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:30.767078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:30.767092 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:30.767181 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:30.768100 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:30.768953 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:30.770021 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:30.770400 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:30.770497 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:30.773531 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:30.773533 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:30.773533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:30.773542 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:30.774898 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:30.775315 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:30.775317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:30.775317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:30.775388 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:30.775763 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:30.775764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:30.775764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:30.775880 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:30.775946 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:30.776198 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 03:28:30.776295 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 03:28:30.776554 392 external_mini_cluster.cc:1428] Setting key 5f9529bb349b71e1e6b25b0b33b2e5cb
I20260927 03:28:30.776597 8027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:30.766270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34037 (local address 127.0.98.1:34019)
0927 03:28:30.766439 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:30.766480 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:30.766975 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:30.767270 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:30.767275 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:30.767671 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:30.767983 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:30.767991 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:30.769058 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:30.769059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:30.769908 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:30.769911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:30.770520 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:30.770521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:30.770680 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:30.771830 (+ 1150us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:30.771839 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:30.773375 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:30.774979 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:30.774985 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:30.774987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:30.775219 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:30.775544 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:30.775544 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:30.775545 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:30.775666 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:30.775940 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:30.775945 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:30.776065 (+ 120us) client_negotiation.cc:769] Sending connection context
0927 03:28:30.776151 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 03:28:30.776347 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
I20260927 03:28:30.777096 8134 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34037
I20260927 03:28:30.777204 8134 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:30.777384 8134 heartbeater.cc:511] Master 127.0.98.62:34037 requested a full tablet report, sending...
I20260927 03:28:30.777920 7931 ts_manager.cc:194] Registered new tserver with Master: 7c403450386c4d7290117df544ead104 (127.0.98.1:45955)
I20260927 03:28:30.778774 7931 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34019
I20260927 03:28:30.784672 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:30.778297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56851 (local address 127.0.98.62:34037)
0927 03:28:30.778331 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:30.778333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:30.778436 (+ 103us) server_negotiation.cc:407] Connection header received
0927 03:28:30.778481 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:30.778481 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:30.778490 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:30.778539 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:30.778942 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:30.779646 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:30.780602 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:30.780810 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:30.780872 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:30.781989 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:30.781990 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:30.781990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:30.781996 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:30.783516 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:30.783914 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:30.783915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:30.783915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:30.783965 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:30.784176 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:30.784177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:30.784177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:30.784283 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:30.784349 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:30.784545 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 03:28:30.784610 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:28:30.784775 8140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:30.778239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34037 (local address 127.0.98.1:56851)
0927 03:28:30.778366 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:30.778377 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:30.778430 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:30.778564 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:30.778565 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:30.778784 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:30.778853 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:30.778855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:30.779740 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:30.779741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:30.780532 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:30.780534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:30.780890 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:30.780891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:30.780961 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:30.781433 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:30.781439 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:30.781903 (+ 464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:30.783613 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:30.783614 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:30.783614 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:30.783842 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:30.784028 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:30.784029 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:30.784029 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:30.784096 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:30.784356 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:30.784358 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:30.784408 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 03:28:30.784521 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 03:28:30.784641 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 03:28:30.803818 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34037
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:30.884055 8143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:30.884253 8143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:30.884276 8143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:30.885404 8143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:30.885437 8143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:30.885457 8143 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:28:30.885474 8143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:28:30.886830 8143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:30.887003 8143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:30.887163 8143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:30.892099 8148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:30.892091 8151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:30.892091 8149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:30.892170 8143 server_base.cc:1034] running on GCE node
I20260927 03:28:30.892498 8143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:30.892746 8143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:30.893890 8143 hybrid_clock.cc:648] HybridClock initialized: now 1790479710893876 us; error 30 us; skew 500 ppm
Sep 27 03:28:30 dist-test-slave-j0hf krb5kdc[7886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479710, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:30.896452 8143 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 03:28:30.896910 8143 webserver.cc:460] Webserver started at http://127.0.98.2:46277/ using document root <none> and password file <none>
I20260927 03:28:30.897147 8143 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:30.897214 8143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:30.897362 8143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:30.898448 8143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-1/data/instance:
uuid: "eb2abbee5aaf45108692df1065604115"
format_stamp: "Formatted at 2026-09-27 03:28:30 on dist-test-slave-j0hf"
server_key: "f57c298eab16fe3129c88e53a5be08a8"
server_key_iv: "c792fbf9ee0979b82c3bba8c61c804d0"
server_key_version: "encryptionkey@0"
I20260927 03:28:30.898865 8143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-1/wal/instance:
uuid: "eb2abbee5aaf45108692df1065604115"
format_stamp: "Formatted at 2026-09-27 03:28:30 on dist-test-slave-j0hf"
server_key: "f57c298eab16fe3129c88e53a5be08a8"
server_key_iv: "c792fbf9ee0979b82c3bba8c61c804d0"
server_key_version: "encryptionkey@0"
I20260927 03:28:30.900264 8143 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:28:30.901031 8158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:30.901196 8143 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:30.901299 8143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "eb2abbee5aaf45108692df1065604115"
format_stamp: "Formatted at 2026-09-27 03:28:30 on dist-test-slave-j0hf"
server_key: "f57c298eab16fe3129c88e53a5be08a8"
server_key_iv: "c792fbf9ee0979b82c3bba8c61c804d0"
server_key_version: "encryptionkey@0"
I20260927 03:28:30.901373 8143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:30.926015 8143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:30.928524 8143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:30.928700 8143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:30.929240 8143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:30.929308 8143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:30.929343 8143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:30.929374 8143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 03:28:30 dist-test-slave-j0hf krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479710, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 03:28:30.936916 8143 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44121
I20260927 03:28:30.937005 8273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44121 every 8 connection(s)
I20260927 03:28:30.937351 8143 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:28:30.938678 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 8143
I20260927 03:28:30.938817 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:28:30.939124 392 external_mini_cluster.cc:1428] Setting key df5603a4813cd41b03e2a4798f942282
I20260927 03:28:30.940079 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:30.929742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47915 (local address 127.0.98.62:34037)
0927 03:28:30.929818 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:30.929820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:30.930021 (+ 201us) server_negotiation.cc:407] Connection header received
0927 03:28:30.930536 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:30.930538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:30.930552 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:30.930638 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:30.931467 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:30.932363 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:30.933633 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:30.933977 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:30.934057 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:30.936735 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:30.936736 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:30.936737 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:30.936745 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:30.938322 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:30.938970 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:30.938971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:30.938971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:30.939029 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:30.939337 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:30.939338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:30.939339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:30.939519 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:30.939608 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:30.939918 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 03:28:30.940007 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 03:28:30.940404 8167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:30.929720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34037 (local address 127.0.98.2:47915)
0927 03:28:30.929881 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:30.929922 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:30.930426 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:30.930692 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:30.930695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:30.930996 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:30.931326 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:30.931333 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:30.932467 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:30.932469 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:30.933526 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:30.933529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:30.934080 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:30.934081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:30.934233 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:30.934832 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:30.934843 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:30.936628 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:30.938416 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:30.938423 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:30.938425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:30.938877 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:30.939104 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:30.939106 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:30.939106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:30.939258 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:30.939600 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:30.939605 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:30.939744 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 03:28:30.939885 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 03:28:30.940106 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":96,"threads_started":1}
I20260927 03:28:30.940920 8274 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34037
I20260927 03:28:30.941035 8274 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:30.941222 8274 heartbeater.cc:511] Master 127.0.98.62:34037 requested a full tablet report, sending...
I20260927 03:28:30.941613 7931 ts_manager.cc:194] Registered new tserver with Master: eb2abbee5aaf45108692df1065604115 (127.0.98.2:44121)
I20260927 03:28:30.942027 7931 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47915
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 03:28:30.949177 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:30.942354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48951 (local address 127.0.98.62:34037)
0927 03:28:30.942847 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:30.942849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:30.942856 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:28:30.942873 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:30.942873 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:30.942881 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:30.942941 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:30.943359 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:30.943892 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:30.945056 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:30.945233 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:30.945279 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:30.946498 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:30.946499 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:30.946499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:30.946505 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:30.947946 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:30.948367 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:30.948368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:30.948368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:30.948417 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:30.948639 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:30.948640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:30.948640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:30.948773 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:30.948821 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:30.949037 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 03:28:30.949110 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471}
I20260927 03:28:30.949288 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:30.942162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34037 (local address 127.0.98.2:48951)
0927 03:28:30.942358 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:30.942371 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:30.942427 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:30.942969 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:30.942971 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:30.943192 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:30.943279 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:30.943280 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:30.944010 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:30.944011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:30.944961 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:30.944964 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:30.945308 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:30.945309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:30.945387 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:30.945867 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:30.945873 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:30.946394 (+ 521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:30.948042 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:30.948044 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:30.948045 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:30.948286 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:30.948490 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:30.948491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:30.948491 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:30.948564 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:30.948840 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:30.948841 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:30.948898 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 03:28:30.949003 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 03:28:30.949124 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 03:28:30.966065 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34037
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:31.047855 8283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:31.048061 8283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:31.048084 8283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:31.049180 8283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:31.049211 8283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:31.049232 8283 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:28:31.049248 8283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:28:31.050740 8283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:31.050913 8283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:31.051079 8283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:28:31.055936 8283 server_base.cc:1034] running on GCE node
W20260927 03:28:31.055912 8289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:31.055948 8288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:31.055920 8291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:31.056354 8283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:31.056555 8283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:31.057724 8283 hybrid_clock.cc:648] HybridClock initialized: now 1790479711057704 us; error 48 us; skew 500 ppm
Sep 27 03:28:31 dist-test-slave-j0hf krb5kdc[7886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479711, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:31.060285 8283 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 03:28:31.060715 8283 webserver.cc:460] Webserver started at http://127.0.98.3:46005/ using document root <none> and password file <none>
I20260927 03:28:31.060899 8283 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:31.060938 8283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:31.061024 8283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:31.062028 8283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-2/data/instance:
uuid: "dfa72a2a427c444caf00c619d9405394"
format_stamp: "Formatted at 2026-09-27 03:28:31 on dist-test-slave-j0hf"
server_key: "2ef3a8f4005103c07425d02861eef874"
server_key_iv: "0ea37c2b9707d68c7074f84af36a3ef4"
server_key_version: "encryptionkey@0"
I20260927 03:28:31.062326 8283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-2/wal/instance:
uuid: "dfa72a2a427c444caf00c619d9405394"
format_stamp: "Formatted at 2026-09-27 03:28:31 on dist-test-slave-j0hf"
server_key: "2ef3a8f4005103c07425d02861eef874"
server_key_iv: "0ea37c2b9707d68c7074f84af36a3ef4"
server_key_version: "encryptionkey@0"
I20260927 03:28:31.063619 8283 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:28:31.064368 8298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:31.064548 8283 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:31.064656 8283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "dfa72a2a427c444caf00c619d9405394"
format_stamp: "Formatted at 2026-09-27 03:28:31 on dist-test-slave-j0hf"
server_key: "2ef3a8f4005103c07425d02861eef874"
server_key_iv: "0ea37c2b9707d68c7074f84af36a3ef4"
server_key_version: "encryptionkey@0"
I20260927 03:28:31.064734 8283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:31.072994 8283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:31.074792 8283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:31.074955 8283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:31.075502 8283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:31.075544 8283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:31.075585 8283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:31.075610 8283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:31.082635 8283 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36683
I20260927 03:28:31.082731 8413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36683 every 8 connection(s)
Sep 27 03:28:31 dist-test-slave-j0hf krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479711, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 03:28:31.083067 8283 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:28:31.086083 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:31.076028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38967 (local address 127.0.98.62:34037)
0927 03:28:31.076102 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:31.076104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:31.076307 (+ 203us) server_negotiation.cc:407] Connection header received
0927 03:28:31.076883 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:31.076884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:31.076897 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:31.076981 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:31.077852 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.078695 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.080370 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.080691 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.080762 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.083681 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:31.083682 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:31.083683 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:31.083689 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:31.084936 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:31.085298 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:31.085299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:31.085299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:31.085348 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:31.085554 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:31.085555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:31.085555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:31.085656 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:31.085705 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:31.085957 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 03:28:31.086019 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 03:28:31.086265 8307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:31.076015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34037 (local address 127.0.98.3:38967)
0927 03:28:31.076175 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:31.076214 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:31.076778 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:31.077068 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:31.077072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:31.077454 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:31.077736 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:31.077743 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.078810 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.078812 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:31.080267 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:31.080283 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.080799 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.080800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:31.080941 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.081606 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:31.081620 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:31.083590 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:31.085007 (+ 1417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:31.085013 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:31.085015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:31.085228 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:31.085404 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:31.085405 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:31.085405 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:31.085501 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:31.085727 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:31.085731 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:31.085842 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 03:28:31.085934 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 03:28:31.086056 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":100,"threads_started":1}
I20260927 03:28:31.086793 8414 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34037
I20260927 03:28:31.086879 8414 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:31.087082 8414 heartbeater.cc:511] Master 127.0.98.62:34037 requested a full tablet report, sending...
I20260927 03:28:31.087479 7931 ts_manager.cc:194] Registered new tserver with Master: dfa72a2a427c444caf00c619d9405394 (127.0.98.3:36683)
I20260927 03:28:31.087975 7931 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38967
I20260927 03:28:31.090842 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 8283
I20260927 03:28:31.090966 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:28:31.091189 392 external_mini_cluster.cc:1428] Setting key 04d982de2a7b29ea5e0ffa024bc4d25e
I20260927 03:28:31.092073 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:28:31.093426 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:31.087773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34037 (local address 127.0.98.3:34881)
0927 03:28:31.087919 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:31.087931 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:31.087983 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:31.088197 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:31.088198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:31.088385 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:31.088480 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:31.088482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.089197 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.089198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:31.089690 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:31.089692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.089996 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.089996 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:31.090042 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.090414 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:31.090418 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:31.090679 (+ 261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:31.092515 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:31.092516 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:31.092516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:31.092674 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:31.092874 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:31.092875 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:31.092875 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:31.092925 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:31.093189 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:31.093190 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:31.093224 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 03:28:31.093281 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 03:28:31.093355 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":97,"threads_started":1}
I20260927 03:28:31.093632 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:31.087860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34881 (local address 127.0.98.62:34037)
0927 03:28:31.088105 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:31.088107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:31.088115 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:28:31.088133 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:31.088133 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:31.088143 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:31.088192 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:31.088567 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.089107 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.089760 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.089934 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.089981 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.090760 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:31.090761 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:31.090762 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:31.090769 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:31.092439 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:31.092746 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:31.092747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:31.092748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:31.092809 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:31.092984 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:31.092985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:31.092985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:31.093127 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:31.093196 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:31.093307 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 03:28:31.093383 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226}
Sep 27 03:28:31 dist-test-slave-j0hf krb5kdc[7886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479711, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 03:28:31 dist-test-slave-j0hf krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:31 dist-test-slave-j0hf krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 03:28:31.114058 7980 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:31.108523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40414 (local address 127.0.98.62:34037)
0927 03:28:31.108583 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:31.108585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:31.108595 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:28:31.108645 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:31.108646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:31.108662 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:31.108727 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:31.109329 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.109822 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.110568 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.110755 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.110841 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.113967 (+ 3126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40414: BlockingRecv error: recv got EOF from 127.0.0.1:40414 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 03:28:31 dist-test-slave-j0hf krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479711, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 03:28:31.121050 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:31.114781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40416 (local address 127.0.98.62:34037)
0927 03:28:31.114854 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:31.114856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:31.114924 (+ 68us) server_negotiation.cc:407] Connection header received
0927 03:28:31.114983 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:31.114983 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:31.114992 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:31.115043 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:31.115533 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.115990 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.116632 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.116800 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.116847 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.118876 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:31.118878 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:31.118878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:31.118884 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:31.120177 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:31.120481 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:31.120481 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:31.120482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:31.120528 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:31.120686 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:31.120687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:31.120687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:31.120779 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:31.120851 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:31.120935 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 03:28:31.120987 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 03:28:31.122529 7931 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:28:31.124462 7931 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:28:31.130856 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:31.126749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44121 (local address 127.0.0.1:56576)
0927 03:28:31.126944 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:31.126986 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:31.127055 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:31.127403 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:31.127407 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:31.127409 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:31.127512 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:31.127514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.128511 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.128512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:31.129620 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:31.129622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.130530 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.130531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:31.130616 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.130631 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 03:28:31.130712 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 03:28:31.130776 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":68,"threads_started":1}
I20260927 03:28:31.131018 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:31.126808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45955 (local address 127.0.0.1:59264)
0927 03:28:31.127087 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:31.127099 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:31.127153 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:31.127752 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:31.127752 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:31.127754 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:31.127802 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:31.127803 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.129065 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.129066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:31.130029 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:31.130031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.130813 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.130814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:31.130870 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.130874 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:28:31.130904 (+ 30us) client_negotiation.cc:240] Negotiation successful
0927 03:28:31.130947 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":28,"thread_start_us":84,"threads_started":1}
I20260927 03:28:31.130884 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:31.126956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56576 (local address 127.0.98.2:44121)
0927 03:28:31.127127 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:31.127132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:31.127142 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:28:31.127187 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:31.127189 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:31.127207 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:31.127331 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:31.127649 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.128426 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.129707 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.130445 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.130514 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.130591 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:31.130733 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 03:28:31.130775 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":72,"threads_started":1}
I20260927 03:28:31.131424 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:31.126918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59264 (local address 127.0.98.1:45955)
0927 03:28:31.127605 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:31.127611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:31.127621 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:28:31.127657 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:31.127659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:31.127675 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:31.127755 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:31.127881 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.128972 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.130118 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.130713 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.131121 (+ 408us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.131225 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:31.131258 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 03:28:31.131325 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":75,"threads_started":1}
I20260927 03:28:31.131488 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:31.126954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36683 (local address 127.0.0.1:36614)
0927 03:28:31.127369 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:31.127379 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:31.127467 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:31.127686 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:31.127686 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:31.127688 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:31.127740 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:31.127741 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.129078 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.129079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:31.130218 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:31.130220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.131222 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.131223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:31.131286 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.131290 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:28:31.131345 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 03:28:31.131384 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":19,"thread_start_us":96,"threads_started":1}
I20260927 03:28:31.131522 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:31.127179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36614 (local address 127.0.98.3:36683)
0927 03:28:31.127371 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:31.127375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:31.127440 (+ 65us) server_negotiation.cc:407] Connection header received
0927 03:28:31.127534 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:31.127536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:31.127552 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:31.127630 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:31.127822 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.128968 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.130314 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.131059 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.131123 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.131190 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:31.131360 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 03:28:31.131396 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":102,"threads_started":1}
I20260927 03:28:31.132009 8208 tablet_service.cc:1467] Processing CreateTablet for tablet 6e4b5bb4f7c3464989dc5afcfff46297 (DEFAULT_TABLE table=test-table [id=b437ef61636b40e1879264d6c59b2130]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:31.132102 8068 tablet_service.cc:1467] Processing CreateTablet for tablet 6e4b5bb4f7c3464989dc5afcfff46297 (DEFAULT_TABLE table=test-table [id=b437ef61636b40e1879264d6c59b2130]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:31.132313 8208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e4b5bb4f7c3464989dc5afcfff46297. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:31.132357 8068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e4b5bb4f7c3464989dc5afcfff46297. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:31.135214 8348 tablet_service.cc:1467] Processing CreateTablet for tablet 6e4b5bb4f7c3464989dc5afcfff46297 (DEFAULT_TABLE table=test-table [id=b437ef61636b40e1879264d6c59b2130]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:31.135546 8442 tablet_bootstrap.cc:492] T 6e4b5bb4f7c3464989dc5afcfff46297 P 7c403450386c4d7290117df544ead104: Bootstrap starting.
I20260927 03:28:31.135632 8348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e4b5bb4f7c3464989dc5afcfff46297. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:31.135799 8443 tablet_bootstrap.cc:492] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115: Bootstrap starting.
I20260927 03:28:31.136303 8442 tablet_bootstrap.cc:654] T 6e4b5bb4f7c3464989dc5afcfff46297 P 7c403450386c4d7290117df544ead104: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:31.136334 8443 tablet_bootstrap.cc:654] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:31.136672 8443 log.cc:824] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:31.136668 8442 log.cc:824] T 6e4b5bb4f7c3464989dc5afcfff46297 P 7c403450386c4d7290117df544ead104: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:31.137410 8442 tablet_bootstrap.cc:492] T 6e4b5bb4f7c3464989dc5afcfff46297 P 7c403450386c4d7290117df544ead104: No bootstrap required, opened a new log
I20260927 03:28:31.137471 8442 ts_tablet_manager.cc:1397] T 6e4b5bb4f7c3464989dc5afcfff46297 P 7c403450386c4d7290117df544ead104: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:28:31.137949 8443 tablet_bootstrap.cc:492] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115: No bootstrap required, opened a new log
I20260927 03:28:31.138023 8443 ts_tablet_manager.cc:1397] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:28:31.138875 8446 tablet_bootstrap.cc:492] T 6e4b5bb4f7c3464989dc5afcfff46297 P dfa72a2a427c444caf00c619d9405394: Bootstrap starting.
I20260927 03:28:31.139504 8446 tablet_bootstrap.cc:654] T 6e4b5bb4f7c3464989dc5afcfff46297 P dfa72a2a427c444caf00c619d9405394: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:31.139503 8442 raft_consensus.cc:348] T 6e4b5bb4f7c3464989dc5afcfff46297 P 7c403450386c4d7290117df544ead104 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.139673 8442 raft_consensus.cc:374] T 6e4b5bb4f7c3464989dc5afcfff46297 P 7c403450386c4d7290117df544ead104 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:31.139719 8442 raft_consensus.cc:729] T 6e4b5bb4f7c3464989dc5afcfff46297 P 7c403450386c4d7290117df544ead104 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c403450386c4d7290117df544ead104, State: Initialized, Role: FOLLOWER
I20260927 03:28:31.139863 8446 log.cc:824] T 6e4b5bb4f7c3464989dc5afcfff46297 P dfa72a2a427c444caf00c619d9405394: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:31.139860 8442 consensus_queue.cc:260] T 6e4b5bb4f7c3464989dc5afcfff46297 P 7c403450386c4d7290117df544ead104 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.140127 8442 ts_tablet_manager.cc:1428] T 6e4b5bb4f7c3464989dc5afcfff46297 P 7c403450386c4d7290117df544ead104: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:28:31.140033 8443 raft_consensus.cc:348] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.140173 8443 raft_consensus.cc:374] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:31.140177 8134 heartbeater.cc:503] Master 127.0.98.62:34037 was elected leader, sending a full tablet report...
I20260927 03:28:31.140213 8443 raft_consensus.cc:729] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2abbee5aaf45108692df1065604115, State: Initialized, Role: FOLLOWER
I20260927 03:28:31.140328 8443 consensus_queue.cc:260] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.140592 8274 heartbeater.cc:503] Master 127.0.98.62:34037 was elected leader, sending a full tablet report...
I20260927 03:28:31.140620 8443 ts_tablet_manager.cc:1428] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 03:28:31.140792 8446 tablet_bootstrap.cc:492] T 6e4b5bb4f7c3464989dc5afcfff46297 P dfa72a2a427c444caf00c619d9405394: No bootstrap required, opened a new log
I20260927 03:28:31.140854 8446 ts_tablet_manager.cc:1397] T 6e4b5bb4f7c3464989dc5afcfff46297 P dfa72a2a427c444caf00c619d9405394: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:28:31.142617 8446 raft_consensus.cc:348] T 6e4b5bb4f7c3464989dc5afcfff46297 P dfa72a2a427c444caf00c619d9405394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.142796 8446 raft_consensus.cc:374] T 6e4b5bb4f7c3464989dc5afcfff46297 P dfa72a2a427c444caf00c619d9405394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:31.142822 8446 raft_consensus.cc:729] T 6e4b5bb4f7c3464989dc5afcfff46297 P dfa72a2a427c444caf00c619d9405394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa72a2a427c444caf00c619d9405394, State: Initialized, Role: FOLLOWER
I20260927 03:28:31.142942 8446 consensus_queue.cc:260] T 6e4b5bb4f7c3464989dc5afcfff46297 P dfa72a2a427c444caf00c619d9405394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.143191 8446 ts_tablet_manager.cc:1428] T 6e4b5bb4f7c3464989dc5afcfff46297 P dfa72a2a427c444caf00c619d9405394: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 03:28:31.143242 8414 heartbeater.cc:503] Master 127.0.98.62:34037 was elected leader, sending a full tablet report...
W20260927 03:28:31.187951 8275 tablet.cc:2369] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:28:31.274179 8135 tablet.cc:2369] T 6e4b5bb4f7c3464989dc5afcfff46297 P 7c403450386c4d7290117df544ead104: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:28:31.297817 8449 raft_consensus.cc:482] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:28:31.297963 8449 raft_consensus.cc:504] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.298460 8449 leader_election.cc:288] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c403450386c4d7290117df544ead104 (127.0.98.1:45955), dfa72a2a427c444caf00c619d9405394 (127.0.98.3:36683)
I20260927 03:28:31.301442 8167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:31.298653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45955 (local address 127.0.98.2:56341)
0927 03:28:31.298744 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:31.298756 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:31.298806 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:31.298986 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:31.298987 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:31.298989 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:31.299063 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:31.299065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.299779 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.299780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:31.300607 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:31.300610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.301149 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.301150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:31.301242 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.301245 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:28:31.301315 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 03:28:31.301362 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 03:28:31.301442 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:31.298718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56341 (local address 127.0.98.1:45955)
0927 03:28:31.298788 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:31.298790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:31.298801 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:28:31.298894 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:31.298895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:31.298911 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:31.298980 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:31.299148 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.299695 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.300703 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.301066 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.301152 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.301163 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:31.301321 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 03:28:31.301364 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 03:28:31.301751 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:31.298799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49155 (local address 127.0.98.3:36683)
0927 03:28:31.298854 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:31.298856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:31.299058 (+ 202us) server_negotiation.cc:407] Connection header received
0927 03:28:31.299143 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:31.299145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:31.299157 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:31.299257 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:31.299449 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.299926 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.300911 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.301297 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.301399 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.301410 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:31.301607 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 03:28:31.301662 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 03:28:31.301797 8088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4b5bb4f7c3464989dc5afcfff46297" candidate_uuid: "eb2abbee5aaf45108692df1065604115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c403450386c4d7290117df544ead104" is_pre_election: true
I20260927 03:28:31.301980 8088 raft_consensus.cc:2454] T 6e4b5bb4f7c3464989dc5afcfff46297 P 7c403450386c4d7290117df544ead104 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2abbee5aaf45108692df1065604115 in term 0.
I20260927 03:28:31.302023 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:31.298659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36683 (local address 127.0.98.2:49155)
0927 03:28:31.298970 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:31.298988 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:31.299072 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:31.299263 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:31.299264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:31.299265 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:31.299345 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:31.299346 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.300037 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.300039 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:31.300795 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:31.300797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.301402 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.301404 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:31.301483 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.301487 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:28:31.301863 (+ 376us) client_negotiation.cc:240] Negotiation successful
0927 03:28:31.301918 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":129,"thread_start_us":124,"threads_started":1}
I20260927 03:28:31.302255 8162 leader_election.cc:302] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c403450386c4d7290117df544ead104, eb2abbee5aaf45108692df1065604115; no voters:
I20260927 03:28:31.302386 8449 raft_consensus.cc:2781] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:28:31.302459 8449 raft_consensus.cc:482] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:28:31.302385 8365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4b5bb4f7c3464989dc5afcfff46297" candidate_uuid: "eb2abbee5aaf45108692df1065604115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa72a2a427c444caf00c619d9405394" is_pre_election: true
I20260927 03:28:31.302481 8449 raft_consensus.cc:3037] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:31.302551 8365 raft_consensus.cc:2454] T 6e4b5bb4f7c3464989dc5afcfff46297 P dfa72a2a427c444caf00c619d9405394 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2abbee5aaf45108692df1065604115 in term 0.
I20260927 03:28:31.303581 8449 raft_consensus.cc:504] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.303717 8449 leader_election.cc:288] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115 [CANDIDATE]: Term 1 election: Requested vote from peers 7c403450386c4d7290117df544ead104 (127.0.98.1:45955), dfa72a2a427c444caf00c619d9405394 (127.0.98.3:36683)
I20260927 03:28:31.303865 8088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4b5bb4f7c3464989dc5afcfff46297" candidate_uuid: "eb2abbee5aaf45108692df1065604115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c403450386c4d7290117df544ead104"
I20260927 03:28:31.303960 8088 raft_consensus.cc:3037] T 6e4b5bb4f7c3464989dc5afcfff46297 P 7c403450386c4d7290117df544ead104 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:31.304024 8365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4b5bb4f7c3464989dc5afcfff46297" candidate_uuid: "eb2abbee5aaf45108692df1065604115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa72a2a427c444caf00c619d9405394"
I20260927 03:28:31.304096 8365 raft_consensus.cc:3037] T 6e4b5bb4f7c3464989dc5afcfff46297 P dfa72a2a427c444caf00c619d9405394 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:31.304867 8088 raft_consensus.cc:2454] T 6e4b5bb4f7c3464989dc5afcfff46297 P 7c403450386c4d7290117df544ead104 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2abbee5aaf45108692df1065604115 in term 1.
I20260927 03:28:31.304986 8365 raft_consensus.cc:2454] T 6e4b5bb4f7c3464989dc5afcfff46297 P dfa72a2a427c444caf00c619d9405394 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2abbee5aaf45108692df1065604115 in term 1.
I20260927 03:28:31.305038 8162 leader_election.cc:302] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c403450386c4d7290117df544ead104, eb2abbee5aaf45108692df1065604115; no voters:
I20260927 03:28:31.305133 8449 raft_consensus.cc:2781] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:31.305311 8449 raft_consensus.cc:686] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115 [term 1 LEADER]: Becoming Leader. State: Replica: eb2abbee5aaf45108692df1065604115, State: Running, Role: LEADER
I20260927 03:28:31.305419 8449 consensus_queue.cc:237] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.306218 7931 catalog_manager.cc:5495] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb2abbee5aaf45108692df1065604115 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "eb2abbee5aaf45108692df1065604115" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } health_report { overall_health: UNKNOWN } } }
I20260927 03:28:31.311499 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:31.309334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56584 (local address 127.0.98.2:44121)
0927 03:28:31.309487 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:31.309489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:31.309498 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:28:31.309529 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:31.309530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:31.309553 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:31.309616 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:28:31.309722 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.310167 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.310854 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.311027 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.311074 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.311195 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:28:31.311281 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:28:31.311337 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:31.311367 (+ 30us) server_negotiation.cc:299] Negotiation successful
0927 03:28:31.311382 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260927 03:28:31.313057 8088 raft_consensus.cc:1261] T 6e4b5bb4f7c3464989dc5afcfff46297 P 7c403450386c4d7290117df544ead104 [term 1 FOLLOWER]: Refusing update from remote peer eb2abbee5aaf45108692df1065604115: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:31.313089 8365 raft_consensus.cc:1261] T 6e4b5bb4f7c3464989dc5afcfff46297 P dfa72a2a427c444caf00c619d9405394 [term 1 FOLLOWER]: Refusing update from remote peer eb2abbee5aaf45108692df1065604115: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:31.313413 8452 consensus_queue.cc:1035] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:31.313542 8449 consensus_queue.cc:1035] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:31.316078 8455 mvcc.cc:205] Tried to move back new op lower bound from 7333804897535893504 to 7333804897507602432. Current Snapshot: MvccSnapshot[applied={T|T < 7333804897535893504}]
I20260927 03:28:31.316351 8456 mvcc.cc:205] Tried to move back new op lower bound from 7333804897535893504 to 7333804897507602432. Current Snapshot: MvccSnapshot[applied={T|T < 7333804897535893504}]
I20260927 03:28:31.316461 8457 mvcc.cc:205] Tried to move back new op lower bound from 7333804897535893504 to 7333804897507602432. Current Snapshot: MvccSnapshot[applied={T|T < 7333804897535893504}]
I20260927 03:28:31.318387 7929 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:28:31.318483 7929 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:28:31.319193 7929 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 456730cb44ed4c2f8cb251842ef9f9bb: Sending DeleteTablet for 3 replicas of tablet 6e4b5bb4f7c3464989dc5afcfff46297
I20260927 03:28:31.319545 8208 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e4b5bb4f7c3464989dc5afcfff46297 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56576
I20260927 03:28:31.319567 8348 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e4b5bb4f7c3464989dc5afcfff46297 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36614
I20260927 03:28:31.319591 8068 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e4b5bb4f7c3464989dc5afcfff46297 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59264
I20260927 03:28:31.319774 8466 tablet_replica.cc:330] T 6e4b5bb4f7c3464989dc5afcfff46297 P 7c403450386c4d7290117df544ead104: stopping tablet replica
I20260927 03:28:31.319870 8466 raft_consensus.cc:2229] T 6e4b5bb4f7c3464989dc5afcfff46297 P 7c403450386c4d7290117df544ead104 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:28:31.319983 8466 raft_consensus.cc:2258] T 6e4b5bb4f7c3464989dc5afcfff46297 P 7c403450386c4d7290117df544ead104 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:31.319975 7929 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:28:31.320125 7929 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:28:31.320338 8467 tablet_replica.cc:330] T 6e4b5bb4f7c3464989dc5afcfff46297 P dfa72a2a427c444caf00c619d9405394: stopping tablet replica
I20260927 03:28:31.320430 8467 raft_consensus.cc:2229] T 6e4b5bb4f7c3464989dc5afcfff46297 P dfa72a2a427c444caf00c619d9405394 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:28:31.320009 8465 tablet_replica.cc:330] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115: stopping tablet replica
I20260927 03:28:31.320590 8465 raft_consensus.cc:2229] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:28:31.320700 8467 raft_consensus.cc:2258] T 6e4b5bb4f7c3464989dc5afcfff46297 P dfa72a2a427c444caf00c619d9405394 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:31.320715 8465 raft_consensus.cc:2258] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:31.321254 8465 ts_tablet_manager.cc:1905] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:31.321244 8068 tablet_service.cc:1467] Processing CreateTablet for tablet 65facd95b3c743679a74678f81b2871c (DEFAULT_TABLE table=test-table [id=f6827da65df743c48926483555f77c7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:31.321358 8068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65facd95b3c743679a74678f81b2871c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:31.321377 8208 tablet_service.cc:1467] Processing CreateTablet for tablet 65facd95b3c743679a74678f81b2871c (DEFAULT_TABLE table=test-table [id=f6827da65df743c48926483555f77c7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:31.321487 8208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65facd95b3c743679a74678f81b2871c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:31.322716 8442 tablet_bootstrap.cc:492] T 65facd95b3c743679a74678f81b2871c P 7c403450386c4d7290117df544ead104: Bootstrap starting.
I20260927 03:28:31.322808 8443 tablet_bootstrap.cc:492] T 65facd95b3c743679a74678f81b2871c P eb2abbee5aaf45108692df1065604115: Bootstrap starting.
I20260927 03:28:31.322870 8348 tablet_service.cc:1467] Processing CreateTablet for tablet 65facd95b3c743679a74678f81b2871c (DEFAULT_TABLE table=test-table [id=f6827da65df743c48926483555f77c7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:31.322979 8348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65facd95b3c743679a74678f81b2871c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:31.323153 8442 tablet_bootstrap.cc:654] T 65facd95b3c743679a74678f81b2871c P 7c403450386c4d7290117df544ead104: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:31.323189 8443 tablet_bootstrap.cc:654] T 65facd95b3c743679a74678f81b2871c P eb2abbee5aaf45108692df1065604115: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:31.323704 8466 ts_tablet_manager.cc:1905] T 6e4b5bb4f7c3464989dc5afcfff46297 P 7c403450386c4d7290117df544ead104: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:31.324245 8446 tablet_bootstrap.cc:492] T 65facd95b3c743679a74678f81b2871c P dfa72a2a427c444caf00c619d9405394: Bootstrap starting.
I20260927 03:28:31.324656 8446 tablet_bootstrap.cc:654] T 65facd95b3c743679a74678f81b2871c P dfa72a2a427c444caf00c619d9405394: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:31.325013 8465 ts_tablet_manager.cc:1918] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:28:31.325090 8465 log.cc:1196] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-1/wal/wals/6e4b5bb4f7c3464989dc5afcfff46297
I20260927 03:28:31.325353 8465 ts_tablet_manager.cc:1939] T 6e4b5bb4f7c3464989dc5afcfff46297 P eb2abbee5aaf45108692df1065604115: Deleting consensus metadata
I20260927 03:28:31.325407 8446 tablet_bootstrap.cc:492] T 65facd95b3c743679a74678f81b2871c P dfa72a2a427c444caf00c619d9405394: No bootstrap required, opened a new log
I20260927 03:28:31.325425 8467 ts_tablet_manager.cc:1905] T 6e4b5bb4f7c3464989dc5afcfff46297 P dfa72a2a427c444caf00c619d9405394: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:31.325448 8446 ts_tablet_manager.cc:1397] T 65facd95b3c743679a74678f81b2871c P dfa72a2a427c444caf00c619d9405394: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 03:28:31.325596 8446 raft_consensus.cc:348] T 65facd95b3c743679a74678f81b2871c P dfa72a2a427c444caf00c619d9405394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.325675 8446 raft_consensus.cc:374] T 65facd95b3c743679a74678f81b2871c P dfa72a2a427c444caf00c619d9405394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:31.325702 8446 raft_consensus.cc:729] T 65facd95b3c743679a74678f81b2871c P dfa72a2a427c444caf00c619d9405394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa72a2a427c444caf00c619d9405394, State: Initialized, Role: FOLLOWER
I20260927 03:28:31.325745 8446 consensus_queue.cc:260] T 65facd95b3c743679a74678f81b2871c P dfa72a2a427c444caf00c619d9405394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.325757 7917 catalog_manager.cc:4841] TS eb2abbee5aaf45108692df1065604115 (127.0.98.2:44121): tablet 6e4b5bb4f7c3464989dc5afcfff46297 (table test-table [id=b437ef61636b40e1879264d6c59b2130]) successfully deleted
I20260927 03:28:31.325837 8446 ts_tablet_manager.cc:1428] T 65facd95b3c743679a74678f81b2871c P dfa72a2a427c444caf00c619d9405394: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:31.325899 8466 ts_tablet_manager.cc:1918] T 6e4b5bb4f7c3464989dc5afcfff46297 P 7c403450386c4d7290117df544ead104: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:28:31.325958 8466 log.cc:1196] T 6e4b5bb4f7c3464989dc5afcfff46297 P 7c403450386c4d7290117df544ead104: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-0/wal/wals/6e4b5bb4f7c3464989dc5afcfff46297
I20260927 03:28:31.326207 8466 ts_tablet_manager.cc:1939] T 6e4b5bb4f7c3464989dc5afcfff46297 P 7c403450386c4d7290117df544ead104: Deleting consensus metadata
I20260927 03:28:31.326591 7918 catalog_manager.cc:4841] TS 7c403450386c4d7290117df544ead104 (127.0.98.1:45955): tablet 6e4b5bb4f7c3464989dc5afcfff46297 (table test-table [id=b437ef61636b40e1879264d6c59b2130]) successfully deleted
I20260927 03:28:31.326920 8467 ts_tablet_manager.cc:1918] T 6e4b5bb4f7c3464989dc5afcfff46297 P dfa72a2a427c444caf00c619d9405394: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:28:31.326990 8467 log.cc:1196] T 6e4b5bb4f7c3464989dc5afcfff46297 P dfa72a2a427c444caf00c619d9405394: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-2/wal/wals/6e4b5bb4f7c3464989dc5afcfff46297
I20260927 03:28:31.327214 8442 tablet_bootstrap.cc:492] T 65facd95b3c743679a74678f81b2871c P 7c403450386c4d7290117df544ead104: No bootstrap required, opened a new log
I20260927 03:28:31.327227 8467 ts_tablet_manager.cc:1939] T 6e4b5bb4f7c3464989dc5afcfff46297 P dfa72a2a427c444caf00c619d9405394: Deleting consensus metadata
I20260927 03:28:31.327267 8442 ts_tablet_manager.cc:1397] T 65facd95b3c743679a74678f81b2871c P 7c403450386c4d7290117df544ead104: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260927 03:28:31.327453 8442 raft_consensus.cc:348] T 65facd95b3c743679a74678f81b2871c P 7c403450386c4d7290117df544ead104 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.327543 8442 raft_consensus.cc:374] T 65facd95b3c743679a74678f81b2871c P 7c403450386c4d7290117df544ead104 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:31.327572 8442 raft_consensus.cc:729] T 65facd95b3c743679a74678f81b2871c P 7c403450386c4d7290117df544ead104 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c403450386c4d7290117df544ead104, State: Initialized, Role: FOLLOWER
I20260927 03:28:31.327608 7917 catalog_manager.cc:4841] TS dfa72a2a427c444caf00c619d9405394 (127.0.98.3:36683): tablet 6e4b5bb4f7c3464989dc5afcfff46297 (table test-table [id=b437ef61636b40e1879264d6c59b2130]) successfully deleted
I20260927 03:28:31.327611 8442 consensus_queue.cc:260] T 65facd95b3c743679a74678f81b2871c P 7c403450386c4d7290117df544ead104 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.327891 8442 ts_tablet_manager.cc:1428] T 65facd95b3c743679a74678f81b2871c P 7c403450386c4d7290117df544ead104: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 03:28:31.328145 8443 tablet_bootstrap.cc:492] T 65facd95b3c743679a74678f81b2871c P eb2abbee5aaf45108692df1065604115: No bootstrap required, opened a new log
I20260927 03:28:31.328186 8443 ts_tablet_manager.cc:1397] T 65facd95b3c743679a74678f81b2871c P eb2abbee5aaf45108692df1065604115: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 03:28:31.328339 8443 raft_consensus.cc:348] T 65facd95b3c743679a74678f81b2871c P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.328408 8443 raft_consensus.cc:374] T 65facd95b3c743679a74678f81b2871c P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:31.328424 8443 raft_consensus.cc:729] T 65facd95b3c743679a74678f81b2871c P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2abbee5aaf45108692df1065604115, State: Initialized, Role: FOLLOWER
I20260927 03:28:31.328495 8443 consensus_queue.cc:260] T 65facd95b3c743679a74678f81b2871c P eb2abbee5aaf45108692df1065604115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.328604 8443 ts_tablet_manager.cc:1428] T 65facd95b3c743679a74678f81b2871c P eb2abbee5aaf45108692df1065604115: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260927 03:28:31.333686 8415 tablet.cc:2369] T 65facd95b3c743679a74678f81b2871c P dfa72a2a427c444caf00c619d9405394: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:28:31.370460 8447 raft_consensus.cc:482] T 65facd95b3c743679a74678f81b2871c P 7c403450386c4d7290117df544ead104 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:28:31.370573 8447 raft_consensus.cc:504] T 65facd95b3c743679a74678f81b2871c P 7c403450386c4d7290117df544ead104 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.370922 8447 leader_election.cc:288] T 65facd95b3c743679a74678f81b2871c P 7c403450386c4d7290117df544ead104 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2abbee5aaf45108692df1065604115 (127.0.98.2:44121), dfa72a2a427c444caf00c619d9405394 (127.0.98.3:36683)
I20260927 03:28:31.374608 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:31.371416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57089 (local address 127.0.98.3:36683)
0927 03:28:31.371488 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:31.371490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:31.371637 (+ 147us) server_negotiation.cc:407] Connection header received
0927 03:28:31.371669 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:31.371670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:31.371678 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:31.371733 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:31.371891 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.372622 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.373857 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.374221 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.374309 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.374323 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:31.374495 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 03:28:31.374534 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 03:28:31.374622 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:31.371365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36683 (local address 127.0.98.1:57089)
0927 03:28:31.371546 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:31.371556 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:31.371624 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:31.371736 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:31.371737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:31.371739 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:31.371809 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:31.371811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.372711 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.372712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:31.373766 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:31.373768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.374334 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.374335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:31.374416 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.374420 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:28:31.374490 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 03:28:31.374534 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":120,"threads_started":1}
I20260927 03:28:31.374661 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:31.371242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46873 (local address 127.0.98.2:44121)
0927 03:28:31.371308 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:31.371311 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:31.371379 (+ 68us) server_negotiation.cc:407] Connection header received
0927 03:28:31.371520 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:31.371521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:31.371531 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:31.371567 (+ 36us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:31.371854 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.372320 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.373767 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.374280 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.374352 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.374363 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:31.374561 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 03:28:31.374595 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 03:28:31.374855 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:31.371171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44121 (local address 127.0.98.1:46873)
0927 03:28:31.371313 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:31.371324 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:31.371410 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:31.371658 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:31.371659 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:31.371661 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:31.371740 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:31.371742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.372408 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.372409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:31.373669 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:31.373671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.374397 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.374398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:31.374478 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.374482 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:28:31.374734 (+ 252us) client_negotiation.cc:240] Negotiation successful
0927 03:28:31.374780 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
I20260927 03:28:31.375052 8365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65facd95b3c743679a74678f81b2871c" candidate_uuid: "7c403450386c4d7290117df544ead104" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa72a2a427c444caf00c619d9405394" is_pre_election: true
I20260927 03:28:31.375144 8365 raft_consensus.cc:2454] T 65facd95b3c743679a74678f81b2871c P dfa72a2a427c444caf00c619d9405394 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c403450386c4d7290117df544ead104 in term 0.
I20260927 03:28:31.375133 8228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65facd95b3c743679a74678f81b2871c" candidate_uuid: "7c403450386c4d7290117df544ead104" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2abbee5aaf45108692df1065604115" is_pre_election: true
I20260927 03:28:31.375247 8228 raft_consensus.cc:2454] T 65facd95b3c743679a74678f81b2871c P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c403450386c4d7290117df544ead104 in term 0.
I20260927 03:28:31.375471 8021 leader_election.cc:302] T 65facd95b3c743679a74678f81b2871c P 7c403450386c4d7290117df544ead104 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c403450386c4d7290117df544ead104, eb2abbee5aaf45108692df1065604115; no voters:
I20260927 03:28:31.375624 8447 raft_consensus.cc:2781] T 65facd95b3c743679a74678f81b2871c P 7c403450386c4d7290117df544ead104 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:28:31.375674 8447 raft_consensus.cc:482] T 65facd95b3c743679a74678f81b2871c P 7c403450386c4d7290117df544ead104 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:28:31.375695 8447 raft_consensus.cc:3037] T 65facd95b3c743679a74678f81b2871c P 7c403450386c4d7290117df544ead104 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:31.376574 8447 raft_consensus.cc:504] T 65facd95b3c743679a74678f81b2871c P 7c403450386c4d7290117df544ead104 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.376700 8447 leader_election.cc:288] T 65facd95b3c743679a74678f81b2871c P 7c403450386c4d7290117df544ead104 [CANDIDATE]: Term 1 election: Requested vote from peers eb2abbee5aaf45108692df1065604115 (127.0.98.2:44121), dfa72a2a427c444caf00c619d9405394 (127.0.98.3:36683)
I20260927 03:28:31.376910 8228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65facd95b3c743679a74678f81b2871c" candidate_uuid: "7c403450386c4d7290117df544ead104" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2abbee5aaf45108692df1065604115"
I20260927 03:28:31.376937 8365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65facd95b3c743679a74678f81b2871c" candidate_uuid: "7c403450386c4d7290117df544ead104" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa72a2a427c444caf00c619d9405394"
I20260927 03:28:31.377003 8365 raft_consensus.cc:3037] T 65facd95b3c743679a74678f81b2871c P dfa72a2a427c444caf00c619d9405394 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:31.377012 8228 raft_consensus.cc:3037] T 65facd95b3c743679a74678f81b2871c P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:31.377722 8365 raft_consensus.cc:2454] T 65facd95b3c743679a74678f81b2871c P dfa72a2a427c444caf00c619d9405394 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c403450386c4d7290117df544ead104 in term 1.
I20260927 03:28:31.377821 8228 raft_consensus.cc:2454] T 65facd95b3c743679a74678f81b2871c P eb2abbee5aaf45108692df1065604115 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c403450386c4d7290117df544ead104 in term 1.
I20260927 03:28:31.377866 8021 leader_election.cc:302] T 65facd95b3c743679a74678f81b2871c P 7c403450386c4d7290117df544ead104 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c403450386c4d7290117df544ead104, dfa72a2a427c444caf00c619d9405394; no voters:
I20260927 03:28:31.377974 8447 raft_consensus.cc:2781] T 65facd95b3c743679a74678f81b2871c P 7c403450386c4d7290117df544ead104 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:31.378100 8447 raft_consensus.cc:686] T 65facd95b3c743679a74678f81b2871c P 7c403450386c4d7290117df544ead104 [term 1 LEADER]: Becoming Leader. State: Replica: 7c403450386c4d7290117df544ead104, State: Running, Role: LEADER
I20260927 03:28:31.378168 8447 consensus_queue.cc:237] T 65facd95b3c743679a74678f81b2871c P 7c403450386c4d7290117df544ead104 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.378892 7931 catalog_manager.cc:5495] T 65facd95b3c743679a74678f81b2871c P 7c403450386c4d7290117df544ead104 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c403450386c4d7290117df544ead104 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7c403450386c4d7290117df544ead104" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } health_report { overall_health: UNKNOWN } } }
Sep 27 03:28:31 dist-test-slave-j0hf krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479710, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 03:28:31.408010 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:31.400253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40418 (local address 127.0.98.62:34037)
0927 03:28:31.400330 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:31.400333 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:31.400562 (+ 229us) server_negotiation.cc:407] Connection header received
0927 03:28:31.400614 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:31.400615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:31.400627 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:31.400685 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:31.401127 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.401872 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.402579 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.402863 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.402963 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.405428 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:31.405429 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:31.405430 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:31.405436 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:31.406807 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:31.407241 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:31.407242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:31.407243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:31.407289 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:31.407521 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:31.407522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:31.407522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:31.407618 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:31.407667 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:31.407857 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 03:28:31.407944 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":49}
I20260927 03:28:31.408130 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:31.400313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34037 (local address 127.0.0.1:40418)
0927 03:28:31.400473 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:31.400483 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:31.400553 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:31.400708 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:31.400709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:31.401002 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:31.401048 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:31.401050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.401973 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.401974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:31.402493 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:31.402494 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.402980 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.402981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:31.403018 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.403614 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:31.403618 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:31.405319 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:31.406903 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:31.406909 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:31.406911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:31.407165 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:31.407356 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:31.407357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:31.407357 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:31.407451 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:31.407687 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:31.407693 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:31.407753 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 03:28:31.407833 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 03:28:31.408033 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":66,"thread_start_us":92,"threads_started":1}
I20260927 03:28:31.410120 7931 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40418:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 03:28:31.410339 7931 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:28:31.411782 8068 tablet_service.cc:1467] Processing CreateTablet for tablet 26d34e0fc5574592aa359ac80e7cb4fd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2d7a36683a26491caed6eb5551e2ce40]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:28:31.411834 8348 tablet_service.cc:1467] Processing CreateTablet for tablet 26d34e0fc5574592aa359ac80e7cb4fd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2d7a36683a26491caed6eb5551e2ce40]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:28:31.411953 8348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26d34e0fc5574592aa359ac80e7cb4fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:31.411954 8068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26d34e0fc5574592aa359ac80e7cb4fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:31.413455 8442 tablet_bootstrap.cc:492] T 26d34e0fc5574592aa359ac80e7cb4fd P 7c403450386c4d7290117df544ead104: Bootstrap starting.
I20260927 03:28:31.413586 8208 tablet_service.cc:1467] Processing CreateTablet for tablet 26d34e0fc5574592aa359ac80e7cb4fd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2d7a36683a26491caed6eb5551e2ce40]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:28:31.413719 8208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26d34e0fc5574592aa359ac80e7cb4fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:31.413878 8442 tablet_bootstrap.cc:654] T 26d34e0fc5574592aa359ac80e7cb4fd P 7c403450386c4d7290117df544ead104: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:31.414230 8446 tablet_bootstrap.cc:492] T 26d34e0fc5574592aa359ac80e7cb4fd P dfa72a2a427c444caf00c619d9405394: Bootstrap starting.
I20260927 03:28:31.414641 8446 tablet_bootstrap.cc:654] T 26d34e0fc5574592aa359ac80e7cb4fd P dfa72a2a427c444caf00c619d9405394: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:31.414885 8443 tablet_bootstrap.cc:492] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115: Bootstrap starting.
I20260927 03:28:31.415287 8443 tablet_bootstrap.cc:654] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:31.415405 8442 tablet_bootstrap.cc:492] T 26d34e0fc5574592aa359ac80e7cb4fd P 7c403450386c4d7290117df544ead104: No bootstrap required, opened a new log
I20260927 03:28:31.415470 8442 ts_tablet_manager.cc:1397] T 26d34e0fc5574592aa359ac80e7cb4fd P 7c403450386c4d7290117df544ead104: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:28:31.415606 8442 raft_consensus.cc:348] T 26d34e0fc5574592aa359ac80e7cb4fd P 7c403450386c4d7290117df544ead104 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.415670 8442 raft_consensus.cc:374] T 26d34e0fc5574592aa359ac80e7cb4fd P 7c403450386c4d7290117df544ead104 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:31.415691 8442 raft_consensus.cc:729] T 26d34e0fc5574592aa359ac80e7cb4fd P 7c403450386c4d7290117df544ead104 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c403450386c4d7290117df544ead104, State: Initialized, Role: FOLLOWER
I20260927 03:28:31.415735 8442 consensus_queue.cc:260] T 26d34e0fc5574592aa359ac80e7cb4fd P 7c403450386c4d7290117df544ead104 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.415822 8442 ts_tablet_manager.cc:1428] T 26d34e0fc5574592aa359ac80e7cb4fd P 7c403450386c4d7290117df544ead104: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:31.415894 8443 tablet_bootstrap.cc:492] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115: No bootstrap required, opened a new log
I20260927 03:28:31.415928 8443 ts_tablet_manager.cc:1397] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 03:28:31.415927 8447 tablet_replica.cc:439] T 26d34e0fc5574592aa359ac80e7cb4fd P 7c403450386c4d7290117df544ead104: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } } }
I20260927 03:28:31.415983 8447 tablet_replica.cc:442] T 26d34e0fc5574592aa359ac80e7cb4fd P 7c403450386c4d7290117df544ead104: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:28:31.416028 8443 raft_consensus.cc:348] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.416072 8443 raft_consensus.cc:374] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:31.416085 8443 raft_consensus.cc:729] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2abbee5aaf45108692df1065604115, State: Initialized, Role: FOLLOWER
I20260927 03:28:31.416146 8443 consensus_queue.cc:260] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.416234 8443 ts_tablet_manager.cc:1428] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:31.416244 8452 tablet_replica.cc:439] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } } }
I20260927 03:28:31.416314 8452 tablet_replica.cc:442] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:28:31.416685 8446 tablet_bootstrap.cc:492] T 26d34e0fc5574592aa359ac80e7cb4fd P dfa72a2a427c444caf00c619d9405394: No bootstrap required, opened a new log
I20260927 03:28:31.416738 8446 ts_tablet_manager.cc:1397] T 26d34e0fc5574592aa359ac80e7cb4fd P dfa72a2a427c444caf00c619d9405394: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260927 03:28:31.416910 8446 raft_consensus.cc:348] T 26d34e0fc5574592aa359ac80e7cb4fd P dfa72a2a427c444caf00c619d9405394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.416987 8446 raft_consensus.cc:374] T 26d34e0fc5574592aa359ac80e7cb4fd P dfa72a2a427c444caf00c619d9405394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:31.417009 8446 raft_consensus.cc:729] T 26d34e0fc5574592aa359ac80e7cb4fd P dfa72a2a427c444caf00c619d9405394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa72a2a427c444caf00c619d9405394, State: Initialized, Role: FOLLOWER
I20260927 03:28:31.417049 8446 consensus_queue.cc:260] T 26d34e0fc5574592aa359ac80e7cb4fd P dfa72a2a427c444caf00c619d9405394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.417140 8446 ts_tablet_manager.cc:1428] T 26d34e0fc5574592aa359ac80e7cb4fd P dfa72a2a427c444caf00c619d9405394: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:31.417222 8450 tablet_replica.cc:439] T 26d34e0fc5574592aa359ac80e7cb4fd P dfa72a2a427c444caf00c619d9405394: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } } }
I20260927 03:28:31.417335 8450 tablet_replica.cc:442] T 26d34e0fc5574592aa359ac80e7cb4fd P dfa72a2a427c444caf00c619d9405394: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:28:31.417433 8452 raft_consensus.cc:482] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:28:31.417480 8452 raft_consensus.cc:504] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.417609 8452 leader_election.cc:288] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c403450386c4d7290117df544ead104 (127.0.98.1:45955), dfa72a2a427c444caf00c619d9405394 (127.0.98.3:36683)
I20260927 03:28:31.417768 8365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d34e0fc5574592aa359ac80e7cb4fd" candidate_uuid: "eb2abbee5aaf45108692df1065604115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa72a2a427c444caf00c619d9405394" is_pre_election: true
I20260927 03:28:31.417758 8088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d34e0fc5574592aa359ac80e7cb4fd" candidate_uuid: "eb2abbee5aaf45108692df1065604115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c403450386c4d7290117df544ead104" is_pre_election: true
I20260927 03:28:31.417833 8365 raft_consensus.cc:2454] T 26d34e0fc5574592aa359ac80e7cb4fd P dfa72a2a427c444caf00c619d9405394 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2abbee5aaf45108692df1065604115 in term 0.
I20260927 03:28:31.417840 8088 raft_consensus.cc:2454] T 26d34e0fc5574592aa359ac80e7cb4fd P 7c403450386c4d7290117df544ead104 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2abbee5aaf45108692df1065604115 in term 0.
I20260927 03:28:31.417980 8161 leader_election.cc:302] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dfa72a2a427c444caf00c619d9405394, eb2abbee5aaf45108692df1065604115; no voters:
I20260927 03:28:31.418073 8452 raft_consensus.cc:2781] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:28:31.418103 8452 raft_consensus.cc:482] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:28:31.418128 8452 raft_consensus.cc:3037] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:31.418819 8452 raft_consensus.cc:504] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.418956 8452 leader_election.cc:288] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115 [CANDIDATE]: Term 1 election: Requested vote from peers 7c403450386c4d7290117df544ead104 (127.0.98.1:45955), dfa72a2a427c444caf00c619d9405394 (127.0.98.3:36683)
I20260927 03:28:31.419134 8365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d34e0fc5574592aa359ac80e7cb4fd" candidate_uuid: "eb2abbee5aaf45108692df1065604115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa72a2a427c444caf00c619d9405394"
I20260927 03:28:31.419162 8088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d34e0fc5574592aa359ac80e7cb4fd" candidate_uuid: "eb2abbee5aaf45108692df1065604115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c403450386c4d7290117df544ead104"
I20260927 03:28:31.419205 8365 raft_consensus.cc:3037] T 26d34e0fc5574592aa359ac80e7cb4fd P dfa72a2a427c444caf00c619d9405394 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:31.419216 8088 raft_consensus.cc:3037] T 26d34e0fc5574592aa359ac80e7cb4fd P 7c403450386c4d7290117df544ead104 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:31.419899 8088 raft_consensus.cc:2454] T 26d34e0fc5574592aa359ac80e7cb4fd P 7c403450386c4d7290117df544ead104 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2abbee5aaf45108692df1065604115 in term 1.
I20260927 03:28:31.419899 8365 raft_consensus.cc:2454] T 26d34e0fc5574592aa359ac80e7cb4fd P dfa72a2a427c444caf00c619d9405394 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2abbee5aaf45108692df1065604115 in term 1.
I20260927 03:28:31.420079 8162 leader_election.cc:302] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c403450386c4d7290117df544ead104, eb2abbee5aaf45108692df1065604115; no voters:
I20260927 03:28:31.420198 8452 raft_consensus.cc:2781] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:31.420243 8452 raft_consensus.cc:686] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115 [term 1 LEADER]: Becoming Leader. State: Replica: eb2abbee5aaf45108692df1065604115, State: Running, Role: LEADER
I20260927 03:28:31.420286 8452 consensus_queue.cc:237] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.420509 8452 tablet_replica.cc:439] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115: TxnStatusTablet state changed. Reason: New leader eb2abbee5aaf45108692df1065604115. Latest consensus state: current_term: 1 leader_uuid: "eb2abbee5aaf45108692df1065604115" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } } }
I20260927 03:28:31.420575 8452 tablet_replica.cc:442] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:28:31.420728 7929 catalog_manager.cc:5495] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb2abbee5aaf45108692df1065604115 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "eb2abbee5aaf45108692df1065604115" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } health_report { overall_health: UNKNOWN } } }
I20260927 03:28:31.420883 8485 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 03:28:31.420984 8485 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 03:28:31.427723 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:31.425358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44121 (local address 127.0.0.1:56590)
0927 03:28:31.425435 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:31.425446 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:31.425493 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:31.425641 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:31.425641 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:31.425642 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:28:31.425687 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:31.425689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.426294 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.426295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:31.426875 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:31.426876 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.427253 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.427254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:31.427325 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.427339 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:28:31.427548 (+ 209us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:28:31.427556 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 03:28:31.427609 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 03:28:31.427656 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 03:28:31.427709 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:31.425442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56590 (local address 127.0.98.2:44121)
0927 03:28:31.425485 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:31.425487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:31.425497 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:28:31.425566 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:31.425567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:31.425579 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:31.425635 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:28:31.425735 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.426209 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.426967 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.427166 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.427251 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.427439 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:28:31.427476 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:28:31.427548 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:31.427613 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 03:28:31.427636 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:28:31.428620 8365 raft_consensus.cc:1261] T 26d34e0fc5574592aa359ac80e7cb4fd P dfa72a2a427c444caf00c619d9405394 [term 1 FOLLOWER]: Refusing update from remote peer eb2abbee5aaf45108692df1065604115: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:31.428669 8088 raft_consensus.cc:1261] T 26d34e0fc5574592aa359ac80e7cb4fd P 7c403450386c4d7290117df544ead104 [term 1 FOLLOWER]: Refusing update from remote peer eb2abbee5aaf45108692df1065604115: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:31.428844 8460 consensus_queue.cc:1035] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:31.428956 8452 consensus_queue.cc:1035] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:31.429560 8450 tablet_replica.cc:439] T 26d34e0fc5574592aa359ac80e7cb4fd P dfa72a2a427c444caf00c619d9405394: TxnStatusTablet state changed. Reason: New leader eb2abbee5aaf45108692df1065604115. Latest consensus state: current_term: 1 leader_uuid: "eb2abbee5aaf45108692df1065604115" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } } }
I20260927 03:28:31.429661 8450 tablet_replica.cc:442] T 26d34e0fc5574592aa359ac80e7cb4fd P dfa72a2a427c444caf00c619d9405394: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:28:31.429641 8447 tablet_replica.cc:439] T 26d34e0fc5574592aa359ac80e7cb4fd P 7c403450386c4d7290117df544ead104: TxnStatusTablet state changed. Reason: New leader eb2abbee5aaf45108692df1065604115. Latest consensus state: current_term: 1 leader_uuid: "eb2abbee5aaf45108692df1065604115" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } } }
I20260927 03:28:31.429723 8447 tablet_replica.cc:442] T 26d34e0fc5574592aa359ac80e7cb4fd P 7c403450386c4d7290117df544ead104: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:28:31.430434 8452 tablet_replica.cc:439] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eb2abbee5aaf45108692df1065604115" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } } }
I20260927 03:28:31.430516 8452 tablet_replica.cc:442] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:28:31.430637 8449 tablet_replica.cc:439] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eb2abbee5aaf45108692df1065604115" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } } }
I20260927 03:28:31.430707 8449 tablet_replica.cc:442] T 26d34e0fc5574592aa359ac80e7cb4fd P eb2abbee5aaf45108692df1065604115: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:28:31.431161 8450 tablet_replica.cc:439] T 26d34e0fc5574592aa359ac80e7cb4fd P dfa72a2a427c444caf00c619d9405394: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eb2abbee5aaf45108692df1065604115" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } } }
I20260927 03:28:31.431244 8450 tablet_replica.cc:442] T 26d34e0fc5574592aa359ac80e7cb4fd P dfa72a2a427c444caf00c619d9405394: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:28:31.431612 8447 tablet_replica.cc:439] T 26d34e0fc5574592aa359ac80e7cb4fd P 7c403450386c4d7290117df544ead104: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eb2abbee5aaf45108692df1065604115" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } } }
I20260927 03:28:31.431695 8447 tablet_replica.cc:442] T 26d34e0fc5574592aa359ac80e7cb4fd P 7c403450386c4d7290117df544ead104: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:28:31.434564 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:31.432177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59272 (local address 127.0.98.1:45955)
0927 03:28:31.432212 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:31.432214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:31.432222 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:28:31.432260 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:31.432260 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:31.432278 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:31.432376 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:28:31.432524 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.432999 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.433751 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.433961 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.434043 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.434210 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:28:31.434319 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:28:31.434401 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:31.434451 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 03:28:31.434471 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 03:28:31.439347 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:31.437095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51559 (local address 127.0.98.2:44121)
0927 03:28:31.437158 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:31.437160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:31.437185 (+ 25us) server_negotiation.cc:407] Connection header received
0927 03:28:31.437250 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:31.437251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:31.437264 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:31.437317 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:28:31.437462 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.437912 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.438643 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.438828 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.438908 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.439072 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:28:31.439127 (+ 55us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:28:31.439190 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:31.439250 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 03:28:31.439268 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 03:28:31.439390 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:31.436960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44121 (local address 127.0.98.1:51559)
0927 03:28:31.437127 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:31.437139 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:31.437202 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:31.437319 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:31.437320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:31.437321 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:28:31.437381 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:31.437382 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.437994 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.437996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:31.438562 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:31.438564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.438916 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.438917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:31.438982 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.438994 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:28:31.439191 (+ 197us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:28:31.439194 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:28:31.439246 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 03:28:31.439284 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":78,"threads_started":1}
I20260927 03:28:31.441780 8365 raft_consensus.cc:1261] T 65facd95b3c743679a74678f81b2871c P dfa72a2a427c444caf00c619d9405394 [term 1 FOLLOWER]: Refusing update from remote peer 7c403450386c4d7290117df544ead104: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:31.441851 8228 raft_consensus.cc:1261] T 65facd95b3c743679a74678f81b2871c P eb2abbee5aaf45108692df1065604115 [term 1 FOLLOWER]: Refusing update from remote peer 7c403450386c4d7290117df544ead104: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:31.442028 8447 consensus_queue.cc:1035] T 65facd95b3c743679a74678f81b2871c P 7c403450386c4d7290117df544ead104 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:31.442160 8473 consensus_queue.cc:1035] T 65facd95b3c743679a74678f81b2871c P 7c403450386c4d7290117df544ead104 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:31.450528 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:31.448044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45955 (local address 127.0.98.2:49347)
0927 03:28:31.448118 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:31.448130 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:31.448179 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:31.448290 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:31.448291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:31.448292 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:28:31.448333 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:31.448335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.448911 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.448912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:31.449681 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:31.449684 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.450084 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.450085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:31.450143 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.450158 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:28:31.450356 (+ 198us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:28:31.450358 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:28:31.450401 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 03:28:31.450468 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 03:28:31.450737 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:31.448098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49347 (local address 127.0.98.1:45955)
0927 03:28:31.448136 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:31.448139 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:31.448172 (+ 33us) server_negotiation.cc:407] Connection header received
0927 03:28:31.448226 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:31.448226 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:31.448244 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:31.448286 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:28:31.448373 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.448826 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.449757 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.449996 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.450086 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.450249 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:28:31.450285 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:28:31.450332 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:31.450651 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 03:28:31.450676 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:28:31.463932 7929 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:28:31.464059 7929 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:28:31.464653 7929 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 456730cb44ed4c2f8cb251842ef9f9bb: Sending DeleteTablet for 3 replicas of tablet 65facd95b3c743679a74678f81b2871c
I20260927 03:28:31.464918 8208 tablet_service.cc:1514] Processing DeleteTablet for tablet 65facd95b3c743679a74678f81b2871c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56576
I20260927 03:28:31.464989 8068 tablet_service.cc:1514] Processing DeleteTablet for tablet 65facd95b3c743679a74678f81b2871c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59264
I20260927 03:28:31.464988 8348 tablet_service.cc:1514] Processing DeleteTablet for tablet 65facd95b3c743679a74678f81b2871c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36614
I20260927 03:28:31.465029 8465 tablet_replica.cc:330] T 65facd95b3c743679a74678f81b2871c P eb2abbee5aaf45108692df1065604115: stopping tablet replica
I20260927 03:28:31.465080 8465 raft_consensus.cc:2229] T 65facd95b3c743679a74678f81b2871c P eb2abbee5aaf45108692df1065604115 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:28:31.465085 8466 tablet_replica.cc:330] T 65facd95b3c743679a74678f81b2871c P 7c403450386c4d7290117df544ead104: stopping tablet replica
I20260927 03:28:31.465121 8465 raft_consensus.cc:2258] T 65facd95b3c743679a74678f81b2871c P eb2abbee5aaf45108692df1065604115 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:31.465130 8466 raft_consensus.cc:2229] T 65facd95b3c743679a74678f81b2871c P 7c403450386c4d7290117df544ead104 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:28:31.465184 8467 tablet_replica.cc:330] T 65facd95b3c743679a74678f81b2871c P dfa72a2a427c444caf00c619d9405394: stopping tablet replica
I20260927 03:28:31.465214 8466 raft_consensus.cc:2258] T 65facd95b3c743679a74678f81b2871c P 7c403450386c4d7290117df544ead104 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:31.465237 8467 raft_consensus.cc:2229] T 65facd95b3c743679a74678f81b2871c P dfa72a2a427c444caf00c619d9405394 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:28:31.465269 8467 raft_consensus.cc:2258] T 65facd95b3c743679a74678f81b2871c P dfa72a2a427c444caf00c619d9405394 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:31.465584 8467 ts_tablet_manager.cc:1905] T 65facd95b3c743679a74678f81b2871c P dfa72a2a427c444caf00c619d9405394: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:31.465584 8466 ts_tablet_manager.cc:1905] T 65facd95b3c743679a74678f81b2871c P 7c403450386c4d7290117df544ead104: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:31.465601 8465 ts_tablet_manager.cc:1905] T 65facd95b3c743679a74678f81b2871c P eb2abbee5aaf45108692df1065604115: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:31.467579 8465 ts_tablet_manager.cc:1918] T 65facd95b3c743679a74678f81b2871c P eb2abbee5aaf45108692df1065604115: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:28:31.467658 8465 log.cc:1196] T 65facd95b3c743679a74678f81b2871c P eb2abbee5aaf45108692df1065604115: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-1/wal/wals/65facd95b3c743679a74678f81b2871c
I20260927 03:28:31.467882 8465 ts_tablet_manager.cc:1939] T 65facd95b3c743679a74678f81b2871c P eb2abbee5aaf45108692df1065604115: Deleting consensus metadata
I20260927 03:28:31.468225 7917 catalog_manager.cc:4841] TS eb2abbee5aaf45108692df1065604115 (127.0.98.2:44121): tablet 65facd95b3c743679a74678f81b2871c (table test-table [id=f6827da65df743c48926483555f77c7d]) successfully deleted
I20260927 03:28:31.470904 8467 ts_tablet_manager.cc:1918] T 65facd95b3c743679a74678f81b2871c P dfa72a2a427c444caf00c619d9405394: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:28:31.470991 8467 log.cc:1196] T 65facd95b3c743679a74678f81b2871c P dfa72a2a427c444caf00c619d9405394: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-2/wal/wals/65facd95b3c743679a74678f81b2871c
I20260927 03:28:31.471196 8467 ts_tablet_manager.cc:1939] T 65facd95b3c743679a74678f81b2871c P dfa72a2a427c444caf00c619d9405394: Deleting consensus metadata
I20260927 03:28:31.471518 8466 ts_tablet_manager.cc:1918] T 65facd95b3c743679a74678f81b2871c P 7c403450386c4d7290117df544ead104: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:28:31.471549 7917 catalog_manager.cc:4841] TS dfa72a2a427c444caf00c619d9405394 (127.0.98.3:36683): tablet 65facd95b3c743679a74678f81b2871c (table test-table [id=f6827da65df743c48926483555f77c7d]) successfully deleted
I20260927 03:28:31.471575 8466 log.cc:1196] T 65facd95b3c743679a74678f81b2871c P 7c403450386c4d7290117df544ead104: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-0/wal/wals/65facd95b3c743679a74678f81b2871c
I20260927 03:28:31.471774 8466 ts_tablet_manager.cc:1939] T 65facd95b3c743679a74678f81b2871c P 7c403450386c4d7290117df544ead104: Deleting consensus metadata
I20260927 03:28:31.472078 7918 catalog_manager.cc:4841] TS 7c403450386c4d7290117df544ead104 (127.0.98.1:45955): tablet 65facd95b3c743679a74678f81b2871c (table test-table [id=f6827da65df743c48926483555f77c7d]) successfully deleted
Sep 27 03:28:31 dist-test-slave-j0hf krb5kdc[7886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 03:28:31 dist-test-slave-j0hf krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:31 dist-test-slave-j0hf krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 03:28:31.496187 7980 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:31.490786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40428 (local address 127.0.98.62:34037)
0927 03:28:31.490840 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:31.490842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:31.490980 (+ 138us) server_negotiation.cc:407] Connection header received
0927 03:28:31.491066 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:31.491067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:31.491079 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:31.491153 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:31.491816 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.492315 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.493052 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.493248 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.493333 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.496128 (+ 2795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40428: BlockingRecv error: recv got EOF from 127.0.0.1:40428 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 27 03:28:31 dist-test-slave-j0hf krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 03:28:31.504510 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:31.497252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40436 (local address 127.0.98.62:34037)
0927 03:28:31.497286 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:31.497288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:31.497403 (+ 115us) server_negotiation.cc:407] Connection header received
0927 03:28:31.497482 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:31.497483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:31.497492 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:31.497564 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:31.498058 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.498522 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.499261 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.499551 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.499625 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.501824 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:31.501825 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:31.501825 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:31.501833 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:31.503325 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:31.503708 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:31.503709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:31.503709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:31.503767 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:31.503966 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:31.503967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:31.503968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:31.504075 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:31.504170 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:31.504326 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 03:28:31.504397 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 03:28:31.506049 7929 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:28:31.506227 7929 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:28:31.507580 8348 tablet_service.cc:1467] Processing CreateTablet for tablet a753f481f0cd4e388de52bec3e728933 (DEFAULT_TABLE table=test-table [id=e16a0482174e410a9597cbb2d9c938fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:31.507608 8068 tablet_service.cc:1467] Processing CreateTablet for tablet a753f481f0cd4e388de52bec3e728933 (DEFAULT_TABLE table=test-table [id=e16a0482174e410a9597cbb2d9c938fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:31.507711 8068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a753f481f0cd4e388de52bec3e728933. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:31.507711 8348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a753f481f0cd4e388de52bec3e728933. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:31.508994 8446 tablet_bootstrap.cc:492] T a753f481f0cd4e388de52bec3e728933 P dfa72a2a427c444caf00c619d9405394: Bootstrap starting.
I20260927 03:28:31.509104 8208 tablet_service.cc:1467] Processing CreateTablet for tablet a753f481f0cd4e388de52bec3e728933 (DEFAULT_TABLE table=test-table [id=e16a0482174e410a9597cbb2d9c938fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:31.509225 8208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a753f481f0cd4e388de52bec3e728933. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:31.509637 8446 tablet_bootstrap.cc:654] T a753f481f0cd4e388de52bec3e728933 P dfa72a2a427c444caf00c619d9405394: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:31.509975 8442 tablet_bootstrap.cc:492] T a753f481f0cd4e388de52bec3e728933 P 7c403450386c4d7290117df544ead104: Bootstrap starting.
I20260927 03:28:31.510355 8442 tablet_bootstrap.cc:654] T a753f481f0cd4e388de52bec3e728933 P 7c403450386c4d7290117df544ead104: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:31.510423 8443 tablet_bootstrap.cc:492] T a753f481f0cd4e388de52bec3e728933 P eb2abbee5aaf45108692df1065604115: Bootstrap starting.
I20260927 03:28:31.510769 8443 tablet_bootstrap.cc:654] T a753f481f0cd4e388de52bec3e728933 P eb2abbee5aaf45108692df1065604115: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:31.510797 8446 tablet_bootstrap.cc:492] T a753f481f0cd4e388de52bec3e728933 P dfa72a2a427c444caf00c619d9405394: No bootstrap required, opened a new log
I20260927 03:28:31.510846 8446 ts_tablet_manager.cc:1397] T a753f481f0cd4e388de52bec3e728933 P dfa72a2a427c444caf00c619d9405394: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:28:31.511001 8446 raft_consensus.cc:348] T a753f481f0cd4e388de52bec3e728933 P dfa72a2a427c444caf00c619d9405394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.511067 8446 raft_consensus.cc:374] T a753f481f0cd4e388de52bec3e728933 P dfa72a2a427c444caf00c619d9405394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:31.511087 8446 raft_consensus.cc:729] T a753f481f0cd4e388de52bec3e728933 P dfa72a2a427c444caf00c619d9405394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa72a2a427c444caf00c619d9405394, State: Initialized, Role: FOLLOWER
I20260927 03:28:31.511137 8446 consensus_queue.cc:260] T a753f481f0cd4e388de52bec3e728933 P dfa72a2a427c444caf00c619d9405394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.511234 8446 ts_tablet_manager.cc:1428] T a753f481f0cd4e388de52bec3e728933 P dfa72a2a427c444caf00c619d9405394: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 03:28:31.511415 8443 tablet_bootstrap.cc:492] T a753f481f0cd4e388de52bec3e728933 P eb2abbee5aaf45108692df1065604115: No bootstrap required, opened a new log
I20260927 03:28:31.511495 8443 ts_tablet_manager.cc:1397] T a753f481f0cd4e388de52bec3e728933 P eb2abbee5aaf45108692df1065604115: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 03:28:31.511615 8443 raft_consensus.cc:348] T a753f481f0cd4e388de52bec3e728933 P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.511682 8443 raft_consensus.cc:374] T a753f481f0cd4e388de52bec3e728933 P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:31.511708 8443 raft_consensus.cc:729] T a753f481f0cd4e388de52bec3e728933 P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2abbee5aaf45108692df1065604115, State: Initialized, Role: FOLLOWER
I20260927 03:28:31.511749 8443 consensus_queue.cc:260] T a753f481f0cd4e388de52bec3e728933 P eb2abbee5aaf45108692df1065604115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.511858 8443 ts_tablet_manager.cc:1428] T a753f481f0cd4e388de52bec3e728933 P eb2abbee5aaf45108692df1065604115: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:31.515836 8442 tablet_bootstrap.cc:492] T a753f481f0cd4e388de52bec3e728933 P 7c403450386c4d7290117df544ead104: No bootstrap required, opened a new log
I20260927 03:28:31.515904 8442 ts_tablet_manager.cc:1397] T a753f481f0cd4e388de52bec3e728933 P 7c403450386c4d7290117df544ead104: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260927 03:28:31.516077 8442 raft_consensus.cc:348] T a753f481f0cd4e388de52bec3e728933 P 7c403450386c4d7290117df544ead104 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.516152 8442 raft_consensus.cc:374] T a753f481f0cd4e388de52bec3e728933 P 7c403450386c4d7290117df544ead104 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:31.516180 8442 raft_consensus.cc:729] T a753f481f0cd4e388de52bec3e728933 P 7c403450386c4d7290117df544ead104 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c403450386c4d7290117df544ead104, State: Initialized, Role: FOLLOWER
I20260927 03:28:31.516222 8442 consensus_queue.cc:260] T a753f481f0cd4e388de52bec3e728933 P 7c403450386c4d7290117df544ead104 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.516310 8442 ts_tablet_manager.cc:1428] T a753f481f0cd4e388de52bec3e728933 P 7c403450386c4d7290117df544ead104: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 03:28:31.748049 8460 raft_consensus.cc:482] T a753f481f0cd4e388de52bec3e728933 P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:28:31.748164 8460 raft_consensus.cc:504] T a753f481f0cd4e388de52bec3e728933 P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.748379 8460 leader_election.cc:288] T a753f481f0cd4e388de52bec3e728933 P eb2abbee5aaf45108692df1065604115 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c403450386c4d7290117df544ead104 (127.0.98.1:45955), dfa72a2a427c444caf00c619d9405394 (127.0.98.3:36683)
I20260927 03:28:31.748617 8365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a753f481f0cd4e388de52bec3e728933" candidate_uuid: "eb2abbee5aaf45108692df1065604115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa72a2a427c444caf00c619d9405394" is_pre_election: true
I20260927 03:28:31.748639 8088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a753f481f0cd4e388de52bec3e728933" candidate_uuid: "eb2abbee5aaf45108692df1065604115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c403450386c4d7290117df544ead104" is_pre_election: true
I20260927 03:28:31.748716 8365 raft_consensus.cc:2454] T a753f481f0cd4e388de52bec3e728933 P dfa72a2a427c444caf00c619d9405394 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2abbee5aaf45108692df1065604115 in term 0.
I20260927 03:28:31.748745 8088 raft_consensus.cc:2454] T a753f481f0cd4e388de52bec3e728933 P 7c403450386c4d7290117df544ead104 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2abbee5aaf45108692df1065604115 in term 0.
I20260927 03:28:31.748871 8161 leader_election.cc:302] T a753f481f0cd4e388de52bec3e728933 P eb2abbee5aaf45108692df1065604115 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dfa72a2a427c444caf00c619d9405394, eb2abbee5aaf45108692df1065604115; no voters:
I20260927 03:28:31.748956 8460 raft_consensus.cc:2781] T a753f481f0cd4e388de52bec3e728933 P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:28:31.749008 8460 raft_consensus.cc:482] T a753f481f0cd4e388de52bec3e728933 P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:28:31.749033 8460 raft_consensus.cc:3037] T a753f481f0cd4e388de52bec3e728933 P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:31.749769 8460 raft_consensus.cc:504] T a753f481f0cd4e388de52bec3e728933 P eb2abbee5aaf45108692df1065604115 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.749928 8460 leader_election.cc:288] T a753f481f0cd4e388de52bec3e728933 P eb2abbee5aaf45108692df1065604115 [CANDIDATE]: Term 1 election: Requested vote from peers 7c403450386c4d7290117df544ead104 (127.0.98.1:45955), dfa72a2a427c444caf00c619d9405394 (127.0.98.3:36683)
I20260927 03:28:31.750058 8088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a753f481f0cd4e388de52bec3e728933" candidate_uuid: "eb2abbee5aaf45108692df1065604115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c403450386c4d7290117df544ead104"
I20260927 03:28:31.750133 8088 raft_consensus.cc:3037] T a753f481f0cd4e388de52bec3e728933 P 7c403450386c4d7290117df544ead104 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:31.750106 8365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a753f481f0cd4e388de52bec3e728933" candidate_uuid: "eb2abbee5aaf45108692df1065604115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa72a2a427c444caf00c619d9405394"
I20260927 03:28:31.750211 8365 raft_consensus.cc:3037] T a753f481f0cd4e388de52bec3e728933 P dfa72a2a427c444caf00c619d9405394 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:31.750723 8088 raft_consensus.cc:2454] T a753f481f0cd4e388de52bec3e728933 P 7c403450386c4d7290117df544ead104 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2abbee5aaf45108692df1065604115 in term 1.
I20260927 03:28:31.750841 8365 raft_consensus.cc:2454] T a753f481f0cd4e388de52bec3e728933 P dfa72a2a427c444caf00c619d9405394 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2abbee5aaf45108692df1065604115 in term 1.
I20260927 03:28:31.750882 8162 leader_election.cc:302] T a753f481f0cd4e388de52bec3e728933 P eb2abbee5aaf45108692df1065604115 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c403450386c4d7290117df544ead104, eb2abbee5aaf45108692df1065604115; no voters:
I20260927 03:28:31.750993 8460 raft_consensus.cc:2781] T a753f481f0cd4e388de52bec3e728933 P eb2abbee5aaf45108692df1065604115 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:31.751073 8460 raft_consensus.cc:686] T a753f481f0cd4e388de52bec3e728933 P eb2abbee5aaf45108692df1065604115 [term 1 LEADER]: Becoming Leader. State: Replica: eb2abbee5aaf45108692df1065604115, State: Running, Role: LEADER
I20260927 03:28:31.751127 8460 consensus_queue.cc:237] T a753f481f0cd4e388de52bec3e728933 P eb2abbee5aaf45108692df1065604115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.751703 7929 catalog_manager.cc:5495] T a753f481f0cd4e388de52bec3e728933 P eb2abbee5aaf45108692df1065604115 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb2abbee5aaf45108692df1065604115 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "eb2abbee5aaf45108692df1065604115" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } health_report { overall_health: UNKNOWN } } }
I20260927 03:28:31.794946 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:31.792570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56602 (local address 127.0.98.2:44121)
0927 03:28:31.792608 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:31.792610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:31.792619 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:28:31.792683 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:31.792684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:31.792694 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:31.792792 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:28:31.792976 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.793463 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.794235 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:31.794426 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:31.794500 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:31.794679 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:28:31.794717 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:28:31.794769 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:31.794836 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 03:28:31.794854 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:28:31.795966 8365 raft_consensus.cc:1261] T a753f481f0cd4e388de52bec3e728933 P dfa72a2a427c444caf00c619d9405394 [term 1 FOLLOWER]: Refusing update from remote peer eb2abbee5aaf45108692df1065604115: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:31.795964 8088 raft_consensus.cc:1261] T a753f481f0cd4e388de52bec3e728933 P 7c403450386c4d7290117df544ead104 [term 1 FOLLOWER]: Refusing update from remote peer eb2abbee5aaf45108692df1065604115: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:31.796247 8452 consensus_queue.cc:1035] T a753f481f0cd4e388de52bec3e728933 P eb2abbee5aaf45108692df1065604115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:31.796346 8460 consensus_queue.cc:1035] T a753f481f0cd4e388de52bec3e728933 P eb2abbee5aaf45108692df1065604115 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:31.799001 7931 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:28:31.799085 7931 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:28:31.799721 7931 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 456730cb44ed4c2f8cb251842ef9f9bb: Sending DeleteTablet for 3 replicas of tablet a753f481f0cd4e388de52bec3e728933
I20260927 03:28:31.799944 8068 tablet_service.cc:1514] Processing DeleteTablet for tablet a753f481f0cd4e388de52bec3e728933 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59264
I20260927 03:28:31.799971 8208 tablet_service.cc:1514] Processing DeleteTablet for tablet a753f481f0cd4e388de52bec3e728933 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56576
I20260927 03:28:31.800041 8466 tablet_replica.cc:330] T a753f481f0cd4e388de52bec3e728933 P 7c403450386c4d7290117df544ead104: stopping tablet replica
I20260927 03:28:31.800099 8466 raft_consensus.cc:2229] T a753f481f0cd4e388de52bec3e728933 P 7c403450386c4d7290117df544ead104 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:28:31.800151 8466 raft_consensus.cc:2258] T a753f481f0cd4e388de52bec3e728933 P 7c403450386c4d7290117df544ead104 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:31.800225 8465 tablet_replica.cc:330] T a753f481f0cd4e388de52bec3e728933 P eb2abbee5aaf45108692df1065604115: stopping tablet replica
I20260927 03:28:31.800189 7931 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 03:28:31.800280 8465 raft_consensus.cc:2229] T a753f481f0cd4e388de52bec3e728933 P eb2abbee5aaf45108692df1065604115 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:28:31.800338 8465 raft_consensus.cc:2258] T a753f481f0cd4e388de52bec3e728933 P eb2abbee5aaf45108692df1065604115 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 03:28:31.800338 7931 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:28:31.800613 8348 tablet_service.cc:1514] Processing DeleteTablet for tablet a753f481f0cd4e388de52bec3e728933 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36614
I20260927 03:28:31.800654 8465 ts_tablet_manager.cc:1905] T a753f481f0cd4e388de52bec3e728933 P eb2abbee5aaf45108692df1065604115: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:31.800707 8467 tablet_replica.cc:330] T a753f481f0cd4e388de52bec3e728933 P dfa72a2a427c444caf00c619d9405394: stopping tablet replica
I20260927 03:28:31.800787 8466 ts_tablet_manager.cc:1905] T a753f481f0cd4e388de52bec3e728933 P 7c403450386c4d7290117df544ead104: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:31.800841 8467 raft_consensus.cc:2229] T a753f481f0cd4e388de52bec3e728933 P dfa72a2a427c444caf00c619d9405394 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:28:31.800930 8467 raft_consensus.cc:2258] T a753f481f0cd4e388de52bec3e728933 P dfa72a2a427c444caf00c619d9405394 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:31.802096 8467 ts_tablet_manager.cc:1905] T a753f481f0cd4e388de52bec3e728933 P dfa72a2a427c444caf00c619d9405394: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:31.802325 8466 ts_tablet_manager.cc:1918] T a753f481f0cd4e388de52bec3e728933 P 7c403450386c4d7290117df544ead104: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:28:31.802368 8465 ts_tablet_manager.cc:1918] T a753f481f0cd4e388de52bec3e728933 P eb2abbee5aaf45108692df1065604115: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:28:31.802382 8466 log.cc:1196] T a753f481f0cd4e388de52bec3e728933 P 7c403450386c4d7290117df544ead104: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-0/wal/wals/a753f481f0cd4e388de52bec3e728933
I20260927 03:28:31.802414 8465 log.cc:1196] T a753f481f0cd4e388de52bec3e728933 P eb2abbee5aaf45108692df1065604115: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-1/wal/wals/a753f481f0cd4e388de52bec3e728933
I20260927 03:28:31.802635 8465 ts_tablet_manager.cc:1939] T a753f481f0cd4e388de52bec3e728933 P eb2abbee5aaf45108692df1065604115: Deleting consensus metadata
I20260927 03:28:31.802654 8466 ts_tablet_manager.cc:1939] T a753f481f0cd4e388de52bec3e728933 P 7c403450386c4d7290117df544ead104: Deleting consensus metadata
I20260927 03:28:31.802879 8348 tablet_service.cc:1467] Processing CreateTablet for tablet 7677541d286c4aecb1c0d680db37addc (DEFAULT_TABLE table=test-table [id=fd2aa658726a4c31b09498e8fc4ede4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:31.802968 8208 tablet_service.cc:1467] Processing CreateTablet for tablet 7677541d286c4aecb1c0d680db37addc (DEFAULT_TABLE table=test-table [id=fd2aa658726a4c31b09498e8fc4ede4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:31.803045 8348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7677541d286c4aecb1c0d680db37addc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:31.803061 8208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7677541d286c4aecb1c0d680db37addc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:31.803480 7917 catalog_manager.cc:4841] TS eb2abbee5aaf45108692df1065604115 (127.0.98.2:44121): tablet a753f481f0cd4e388de52bec3e728933 (table test-table [id=e16a0482174e410a9597cbb2d9c938fe]) successfully deleted
I20260927 03:28:31.803834 8467 ts_tablet_manager.cc:1918] T a753f481f0cd4e388de52bec3e728933 P dfa72a2a427c444caf00c619d9405394: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:28:31.803896 8467 log.cc:1196] T a753f481f0cd4e388de52bec3e728933 P dfa72a2a427c444caf00c619d9405394: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-2/wal/wals/a753f481f0cd4e388de52bec3e728933
I20260927 03:28:31.804129 8467 ts_tablet_manager.cc:1939] T a753f481f0cd4e388de52bec3e728933 P dfa72a2a427c444caf00c619d9405394: Deleting consensus metadata
I20260927 03:28:31.804206 7918 catalog_manager.cc:4841] TS 7c403450386c4d7290117df544ead104 (127.0.98.1:45955): tablet a753f481f0cd4e388de52bec3e728933 (table test-table [id=e16a0482174e410a9597cbb2d9c938fe]) successfully deleted
I20260927 03:28:31.804458 8068 tablet_service.cc:1467] Processing CreateTablet for tablet 7677541d286c4aecb1c0d680db37addc (DEFAULT_TABLE table=test-table [id=fd2aa658726a4c31b09498e8fc4ede4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:31.804500 8446 tablet_bootstrap.cc:492] T 7677541d286c4aecb1c0d680db37addc P dfa72a2a427c444caf00c619d9405394: Bootstrap starting.
I20260927 03:28:31.804563 8068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7677541d286c4aecb1c0d680db37addc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:31.804862 8446 tablet_bootstrap.cc:654] T 7677541d286c4aecb1c0d680db37addc P dfa72a2a427c444caf00c619d9405394: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:31.805465 8446 tablet_bootstrap.cc:492] T 7677541d286c4aecb1c0d680db37addc P dfa72a2a427c444caf00c619d9405394: No bootstrap required, opened a new log
I20260927 03:28:31.805495 8446 ts_tablet_manager.cc:1397] T 7677541d286c4aecb1c0d680db37addc P dfa72a2a427c444caf00c619d9405394: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 03:28:31.805601 8446 raft_consensus.cc:348] T 7677541d286c4aecb1c0d680db37addc P dfa72a2a427c444caf00c619d9405394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.805644 8446 raft_consensus.cc:374] T 7677541d286c4aecb1c0d680db37addc P dfa72a2a427c444caf00c619d9405394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:31.805627 8442 tablet_bootstrap.cc:492] T 7677541d286c4aecb1c0d680db37addc P 7c403450386c4d7290117df544ead104: Bootstrap starting.
I20260927 03:28:31.805657 8446 raft_consensus.cc:729] T 7677541d286c4aecb1c0d680db37addc P dfa72a2a427c444caf00c619d9405394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa72a2a427c444caf00c619d9405394, State: Initialized, Role: FOLLOWER
I20260927 03:28:31.805653 8443 tablet_bootstrap.cc:492] T 7677541d286c4aecb1c0d680db37addc P eb2abbee5aaf45108692df1065604115: Bootstrap starting.
I20260927 03:28:31.805685 8446 consensus_queue.cc:260] T 7677541d286c4aecb1c0d680db37addc P dfa72a2a427c444caf00c619d9405394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.805788 8446 ts_tablet_manager.cc:1428] T 7677541d286c4aecb1c0d680db37addc P dfa72a2a427c444caf00c619d9405394: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:31.805917 8443 tablet_bootstrap.cc:654] T 7677541d286c4aecb1c0d680db37addc P eb2abbee5aaf45108692df1065604115: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:31.805927 8442 tablet_bootstrap.cc:654] T 7677541d286c4aecb1c0d680db37addc P 7c403450386c4d7290117df544ead104: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:31.806309 7917 catalog_manager.cc:4841] TS dfa72a2a427c444caf00c619d9405394 (127.0.98.3:36683): tablet a753f481f0cd4e388de52bec3e728933 (table test-table [id=e16a0482174e410a9597cbb2d9c938fe]) successfully deleted
I20260927 03:28:31.806483 8443 tablet_bootstrap.cc:492] T 7677541d286c4aecb1c0d680db37addc P eb2abbee5aaf45108692df1065604115: No bootstrap required, opened a new log
I20260927 03:28:31.806524 8443 ts_tablet_manager.cc:1397] T 7677541d286c4aecb1c0d680db37addc P eb2abbee5aaf45108692df1065604115: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 03:28:31.806617 8442 tablet_bootstrap.cc:492] T 7677541d286c4aecb1c0d680db37addc P 7c403450386c4d7290117df544ead104: No bootstrap required, opened a new log
I20260927 03:28:31.806653 8442 ts_tablet_manager.cc:1397] T 7677541d286c4aecb1c0d680db37addc P 7c403450386c4d7290117df544ead104: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 03:28:31.806620 8443 raft_consensus.cc:348] T 7677541d286c4aecb1c0d680db37addc P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.806686 8443 raft_consensus.cc:374] T 7677541d286c4aecb1c0d680db37addc P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:31.806708 8443 raft_consensus.cc:729] T 7677541d286c4aecb1c0d680db37addc P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2abbee5aaf45108692df1065604115, State: Initialized, Role: FOLLOWER
I20260927 03:28:31.806752 8443 consensus_queue.cc:260] T 7677541d286c4aecb1c0d680db37addc P eb2abbee5aaf45108692df1065604115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.806753 8442 raft_consensus.cc:348] T 7677541d286c4aecb1c0d680db37addc P 7c403450386c4d7290117df544ead104 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.806813 8442 raft_consensus.cc:374] T 7677541d286c4aecb1c0d680db37addc P 7c403450386c4d7290117df544ead104 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:31.806833 8442 raft_consensus.cc:729] T 7677541d286c4aecb1c0d680db37addc P 7c403450386c4d7290117df544ead104 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c403450386c4d7290117df544ead104, State: Initialized, Role: FOLLOWER
I20260927 03:28:31.806834 8443 ts_tablet_manager.cc:1428] T 7677541d286c4aecb1c0d680db37addc P eb2abbee5aaf45108692df1065604115: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:31.806880 8442 consensus_queue.cc:260] T 7677541d286c4aecb1c0d680db37addc P 7c403450386c4d7290117df544ead104 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:31.806990 8442 ts_tablet_manager.cc:1428] T 7677541d286c4aecb1c0d680db37addc P 7c403450386c4d7290117df544ead104: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:32.047962 8450 raft_consensus.cc:482] T 7677541d286c4aecb1c0d680db37addc P dfa72a2a427c444caf00c619d9405394 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:28:32.048085 8450 raft_consensus.cc:504] T 7677541d286c4aecb1c0d680db37addc P dfa72a2a427c444caf00c619d9405394 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:32.048494 8450 leader_election.cc:288] T 7677541d286c4aecb1c0d680db37addc P dfa72a2a427c444caf00c619d9405394 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c403450386c4d7290117df544ead104 (127.0.98.1:45955), eb2abbee5aaf45108692df1065604115 (127.0.98.2:44121)
I20260927 03:28:32.051532 8522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:32.048760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55695 (local address 127.0.98.1:45955)
0927 03:28:32.048879 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:32.048881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:32.048892 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:28:32.048963 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:32.048964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:32.048976 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:32.049073 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:32.049251 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:32.049735 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:32.050737 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:32.051129 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:32.051216 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:32.051229 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:32.051394 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 03:28:32.051465 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":50,"threads_started":1}
I20260927 03:28:32.051537 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:32.048669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45955 (local address 127.0.98.3:55695)
0927 03:28:32.048815 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:32.048827 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:32.048888 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:32.049077 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:32.049078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:32.049080 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:32.049178 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:32.049179 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:32.049824 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:32.049825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:32.050643 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:32.050645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:32.051216 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:32.051217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:32.051295 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:32.051300 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:28:32.051378 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 03:28:32.051424 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":45,"thread_start_us":80,"threads_started":1}
I20260927 03:28:32.051615 8521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:32.048677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44121 (local address 127.0.98.3:50171)
0927 03:28:32.048839 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:32.048850 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:32.048917 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:32.049072 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:32.049073 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:32.049075 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:32.049169 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:32.049171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:32.049806 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:32.049807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:32.050635 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:32.050637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:32.051292 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:32.051294 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:32.051355 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:32.051358 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:28:32.051451 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 03:28:32.051496 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":31,"thread_start_us":62,"threads_started":1}
I20260927 03:28:32.051601 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:32.048803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50171 (local address 127.0.98.2:44121)
0927 03:28:32.048858 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:32.048860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:32.048902 (+ 42us) server_negotiation.cc:407] Connection header received
0927 03:28:32.048990 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:32.048990 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:32.048999 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:32.049070 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:32.049244 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:32.049716 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:32.050745 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:32.051177 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:32.051250 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:32.051262 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:32.051474 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 03:28:32.051509 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 03:28:32.051846 8228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7677541d286c4aecb1c0d680db37addc" candidate_uuid: "dfa72a2a427c444caf00c619d9405394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2abbee5aaf45108692df1065604115" is_pre_election: true
I20260927 03:28:32.051940 8228 raft_consensus.cc:2454] T 7677541d286c4aecb1c0d680db37addc P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa72a2a427c444caf00c619d9405394 in term 0.
I20260927 03:28:32.052062 8088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7677541d286c4aecb1c0d680db37addc" candidate_uuid: "dfa72a2a427c444caf00c619d9405394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c403450386c4d7290117df544ead104" is_pre_election: true
I20260927 03:28:32.052160 8088 raft_consensus.cc:2454] T 7677541d286c4aecb1c0d680db37addc P 7c403450386c4d7290117df544ead104 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa72a2a427c444caf00c619d9405394 in term 0.
I20260927 03:28:32.052341 8301 leader_election.cc:302] T 7677541d286c4aecb1c0d680db37addc P dfa72a2a427c444caf00c619d9405394 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dfa72a2a427c444caf00c619d9405394, eb2abbee5aaf45108692df1065604115; no voters:
I20260927 03:28:32.052501 8450 raft_consensus.cc:2781] T 7677541d286c4aecb1c0d680db37addc P dfa72a2a427c444caf00c619d9405394 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:28:32.052563 8450 raft_consensus.cc:482] T 7677541d286c4aecb1c0d680db37addc P dfa72a2a427c444caf00c619d9405394 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:28:32.052596 8450 raft_consensus.cc:3037] T 7677541d286c4aecb1c0d680db37addc P dfa72a2a427c444caf00c619d9405394 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:32.053282 8450 raft_consensus.cc:504] T 7677541d286c4aecb1c0d680db37addc P dfa72a2a427c444caf00c619d9405394 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:32.053445 8450 leader_election.cc:288] T 7677541d286c4aecb1c0d680db37addc P dfa72a2a427c444caf00c619d9405394 [CANDIDATE]: Term 1 election: Requested vote from peers 7c403450386c4d7290117df544ead104 (127.0.98.1:45955), eb2abbee5aaf45108692df1065604115 (127.0.98.2:44121)
I20260927 03:28:32.053586 8228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7677541d286c4aecb1c0d680db37addc" candidate_uuid: "dfa72a2a427c444caf00c619d9405394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2abbee5aaf45108692df1065604115"
I20260927 03:28:32.053598 8088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7677541d286c4aecb1c0d680db37addc" candidate_uuid: "dfa72a2a427c444caf00c619d9405394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c403450386c4d7290117df544ead104"
I20260927 03:28:32.053664 8228 raft_consensus.cc:3037] T 7677541d286c4aecb1c0d680db37addc P eb2abbee5aaf45108692df1065604115 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:32.053686 8088 raft_consensus.cc:3037] T 7677541d286c4aecb1c0d680db37addc P 7c403450386c4d7290117df544ead104 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:32.054275 8088 raft_consensus.cc:2454] T 7677541d286c4aecb1c0d680db37addc P 7c403450386c4d7290117df544ead104 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa72a2a427c444caf00c619d9405394 in term 1.
I20260927 03:28:32.054281 8228 raft_consensus.cc:2454] T 7677541d286c4aecb1c0d680db37addc P eb2abbee5aaf45108692df1065604115 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa72a2a427c444caf00c619d9405394 in term 1.
I20260927 03:28:32.054461 8302 leader_election.cc:302] T 7677541d286c4aecb1c0d680db37addc P dfa72a2a427c444caf00c619d9405394 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c403450386c4d7290117df544ead104, dfa72a2a427c444caf00c619d9405394; no voters:
I20260927 03:28:32.054562 8450 raft_consensus.cc:2781] T 7677541d286c4aecb1c0d680db37addc P dfa72a2a427c444caf00c619d9405394 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:32.054669 8450 raft_consensus.cc:686] T 7677541d286c4aecb1c0d680db37addc P dfa72a2a427c444caf00c619d9405394 [term 1 LEADER]: Becoming Leader. State: Replica: dfa72a2a427c444caf00c619d9405394, State: Running, Role: LEADER
I20260927 03:28:32.054737 8450 consensus_queue.cc:237] T 7677541d286c4aecb1c0d680db37addc P dfa72a2a427c444caf00c619d9405394 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } }
I20260927 03:28:32.055507 7929 catalog_manager.cc:5495] T 7677541d286c4aecb1c0d680db37addc P dfa72a2a427c444caf00c619d9405394 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfa72a2a427c444caf00c619d9405394 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "dfa72a2a427c444caf00c619d9405394" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa72a2a427c444caf00c619d9405394" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36683 } health_report { overall_health: HEALTHY } } }
I20260927 03:28:32.093452 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:32.090894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36626 (local address 127.0.98.3:36683)
0927 03:28:32.091126 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:32.091128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:32.091134 (+ 6us) server_negotiation.cc:407] Connection header received
0927 03:28:32.091164 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:32.091165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:32.091181 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:32.091269 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:28:32.091393 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:32.091915 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:32.092650 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:32.092845 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:32.092915 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:32.093081 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:28:32.093192 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:28:32.093271 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:32.093313 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 03:28:32.093333 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":163,"threads_started":1}
I20260927 03:28:32.098055 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:32.095953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40525 (local address 127.0.98.2:44121)
0927 03:28:32.096006 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:32.096009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:32.096023 (+ 14us) server_negotiation.cc:407] Connection header received
0927 03:28:32.096043 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:32.096043 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:32.096052 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:32.096096 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:28:32.096250 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:32.096724 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:32.097476 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:32.097656 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:32.097704 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:32.097852 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:28:32.097876 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:28:32.097913 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:32.097981 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 03:28:32.097999 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 03:28:32.098067 8528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:32.095783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44121 (local address 127.0.98.3:40525)
0927 03:28:32.095937 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:32.095948 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:32.095996 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:32.096103 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:32.096103 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:32.096104 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:28:32.096182 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:32.096184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:32.096794 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:32.096794 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:32.097404 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:32.097406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:32.097731 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:32.097731 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:32.097773 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:32.097785 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:28:32.097933 (+ 148us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:28:32.097934 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 03:28:32.097969 (+ 35us) client_negotiation.cc:240] Negotiation successful
0927 03:28:32.098001 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260927 03:28:32.100369 8228 raft_consensus.cc:1261] T 7677541d286c4aecb1c0d680db37addc P eb2abbee5aaf45108692df1065604115 [term 1 FOLLOWER]: Refusing update from remote peer dfa72a2a427c444caf00c619d9405394: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:32.100452 8088 raft_consensus.cc:1261] T 7677541d286c4aecb1c0d680db37addc P 7c403450386c4d7290117df544ead104 [term 1 FOLLOWER]: Refusing update from remote peer dfa72a2a427c444caf00c619d9405394: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:32.100673 8523 consensus_queue.cc:1035] T 7677541d286c4aecb1c0d680db37addc P dfa72a2a427c444caf00c619d9405394 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2abbee5aaf45108692df1065604115" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:32.100811 8450 consensus_queue.cc:1035] T 7677541d286c4aecb1c0d680db37addc P dfa72a2a427c444caf00c619d9405394 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c403450386c4d7290117df544ead104" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:32.107795 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:32.105426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47517 (local address 127.0.98.3:36683)
0927 03:28:32.105468 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:32.105470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:32.105535 (+ 65us) server_negotiation.cc:407] Connection header received
0927 03:28:32.105600 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:32.105601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:32.105612 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:32.105664 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:28:32.105789 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:32.106258 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:32.107015 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:32.107278 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:32.107349 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:32.107546 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:28:32.107581 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:28:32.107638 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:32.107696 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 03:28:32.107719 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:28:32.107822 8533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:32.105391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36683 (local address 127.0.98.2:47517)
0927 03:28:32.105481 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:32.105492 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:32.105549 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:32.105697 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:32.105698 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:32.105699 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:28:32.105747 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:32.105748 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:32.106345 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:32.106345 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:32.106943 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:32.106944 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:32.107363 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:32.107364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:32.107452 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:32.107464 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:28:32.107647 (+ 183us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:28:32.107649 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:28:32.107695 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 03:28:32.107763 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"thread_start_us":53,"threads_started":1}
I20260927 03:28:32.114910 7929 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:28:32.115028 7929 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:28:32.115803 7929 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 456730cb44ed4c2f8cb251842ef9f9bb: Sending DeleteTablet for 3 replicas of tablet 7677541d286c4aecb1c0d680db37addc
I20260927 03:28:32.116066 8208 tablet_service.cc:1514] Processing DeleteTablet for tablet 7677541d286c4aecb1c0d680db37addc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56576
I20260927 03:28:32.116084 8348 tablet_service.cc:1514] Processing DeleteTablet for tablet 7677541d286c4aecb1c0d680db37addc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36614
I20260927 03:28:32.116084 8068 tablet_service.cc:1514] Processing DeleteTablet for tablet 7677541d286c4aecb1c0d680db37addc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59264
I20260927 03:28:32.116178 8466 tablet_replica.cc:330] T 7677541d286c4aecb1c0d680db37addc P 7c403450386c4d7290117df544ead104: stopping tablet replica
I20260927 03:28:32.116181 8465 tablet_replica.cc:330] T 7677541d286c4aecb1c0d680db37addc P eb2abbee5aaf45108692df1065604115: stopping tablet replica
I20260927 03:28:32.116232 8466 raft_consensus.cc:2229] T 7677541d286c4aecb1c0d680db37addc P 7c403450386c4d7290117df544ead104 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:28:32.116256 8465 raft_consensus.cc:2229] T 7677541d286c4aecb1c0d680db37addc P eb2abbee5aaf45108692df1065604115 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:28:32.116269 8466 raft_consensus.cc:2258] T 7677541d286c4aecb1c0d680db37addc P 7c403450386c4d7290117df544ead104 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:32.116294 8465 raft_consensus.cc:2258] T 7677541d286c4aecb1c0d680db37addc P eb2abbee5aaf45108692df1065604115 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:32.116365 8467 tablet_replica.cc:330] T 7677541d286c4aecb1c0d680db37addc P dfa72a2a427c444caf00c619d9405394: stopping tablet replica
I20260927 03:28:32.116420 8467 raft_consensus.cc:2229] T 7677541d286c4aecb1c0d680db37addc P dfa72a2a427c444caf00c619d9405394 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:28:32.116513 8467 raft_consensus.cc:2258] T 7677541d286c4aecb1c0d680db37addc P dfa72a2a427c444caf00c619d9405394 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:32.116751 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 8003
I20260927 03:28:32.117393 8465 ts_tablet_manager.cc:1905] T 7677541d286c4aecb1c0d680db37addc P eb2abbee5aaf45108692df1065604115: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:32.119248 8467 ts_tablet_manager.cc:1905] T 7677541d286c4aecb1c0d680db37addc P dfa72a2a427c444caf00c619d9405394: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:32.119258 8465 ts_tablet_manager.cc:1918] T 7677541d286c4aecb1c0d680db37addc P eb2abbee5aaf45108692df1065604115: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:28:32.119318 8465 log.cc:1196] T 7677541d286c4aecb1c0d680db37addc P eb2abbee5aaf45108692df1065604115: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-1/wal/wals/7677541d286c4aecb1c0d680db37addc
I20260927 03:28:32.119587 8465 ts_tablet_manager.cc:1939] T 7677541d286c4aecb1c0d680db37addc P eb2abbee5aaf45108692df1065604115: Deleting consensus metadata
I20260927 03:28:32.119962 7917 catalog_manager.cc:4841] TS eb2abbee5aaf45108692df1065604115 (127.0.98.2:44121): tablet 7677541d286c4aecb1c0d680db37addc (table test-table [id=fd2aa658726a4c31b09498e8fc4ede4a]) successfully deleted
I20260927 03:28:32.120760 8467 ts_tablet_manager.cc:1918] T 7677541d286c4aecb1c0d680db37addc P dfa72a2a427c444caf00c619d9405394: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:28:32.120841 8467 log.cc:1196] T 7677541d286c4aecb1c0d680db37addc P dfa72a2a427c444caf00c619d9405394: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790479675865103-392-0/minicluster-data/ts-2/wal/wals/7677541d286c4aecb1c0d680db37addc
I20260927 03:28:32.121049 8467 ts_tablet_manager.cc:1939] T 7677541d286c4aecb1c0d680db37addc P dfa72a2a427c444caf00c619d9405394: Deleting consensus metadata
I20260927 03:28:32.121380 7917 catalog_manager.cc:4841] TS dfa72a2a427c444caf00c619d9405394 (127.0.98.3:36683): tablet 7677541d286c4aecb1c0d680db37addc (table test-table [id=fd2aa658726a4c31b09498e8fc4ede4a]) successfully deleted
W20260927 03:28:32.124943 7918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45955 (error 108)
I20260927 03:28:32.125224 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 8143
I20260927 03:28:32.125993 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:32.125347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45955 (local address 127.0.0.1:59276)
0927 03:28:32.125891 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:32.125941 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":39,"threads_started":1}
W20260927 03:28:32.126886 7918 catalog_manager.cc:4584] TS 7c403450386c4d7290117df544ead104 (127.0.98.1:45955): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7677541d286c4aecb1c0d680db37addc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45955: connect: Connection refused (error 111)
I20260927 03:28:32.133375 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 8283
I20260927 03:28:32.138182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 7899
2026-09-27T03:28:32Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2855 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:28:32 dist-test-slave-j0hf krb5kdc[8540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:28:32 dist-test-slave-j0hf krb5kdc[8540](info): set up 2 sockets
Sep 27 03:28:32 dist-test-slave-j0hf krb5kdc[8540](info): commencing operation
krb5kdc: starting...
W20260927 03:28:33.167980 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:28:33 dist-test-slave-j0hf krb5kdc[8540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:28:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:28:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:28:33.289980 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46007
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:33.371492 8553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:33.371762 8553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:33.371803 8553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:33.373059 8553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:28:33.373138 8553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:33.373162 8553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:33.373183 8553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:28:33.373209 8553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:28:33.374847 8553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:33.375147 8553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:33.375388 8553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:33.380206 8558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:33.380193 8561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:33.380401 8553 server_base.cc:1034] running on GCE node
W20260927 03:28:33.380268 8559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:33.380751 8553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:33.380988 8553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:33.382129 8553 hybrid_clock.cc:648] HybridClock initialized: now 1790479713382116 us; error 31 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T03:28:33Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1249 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:28:33 dist-test-slave-j0hf krb5kdc[8566](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:28:33 dist-test-slave-j0hf krb5kdc[8566](info): set up 2 sockets
Sep 27 03:28:33 dist-test-slave-j0hf krb5kdc[8566](info): commencing operation
krb5kdc: starting...
W20260927 03:28:34.460054 392 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:28:34 dist-test-slave-j0hf krb5kdc[8566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:28:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:28:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:28:34.578433 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36343
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:34.658540 8579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:34.658788 8579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:34.658824 8579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:34.660055 8579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:28:34.660120 8579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:34.660146 8579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:34.660174 8579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:28:34.660207 8579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:28:34.661680 8579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:34.661952 8579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:34.662187 8579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:34.666829 8585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:34.666857 8587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:34.666913 8584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:34.666970 8579 server_base.cc:1034] running on GCE node
I20260927 03:28:34.667256 8579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:34.667577 8579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:34.668732 8579 hybrid_clock.cc:648] HybridClock initialized: now 1790479714668709 us; error 55 us; skew 500 ppm
Sep 27 03:28:34 dist-test-slave-j0hf krb5kdc[8566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:34.675360 8579 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:28:34.675799 8579 webserver.cc:460] Webserver started at http://127.0.98.62:37867/ using document root <none> and password file <none>
I20260927 03:28:34.676025 8579 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:34.676092 8579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:34.676240 8579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:34.677318 8579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "be01327a73c94746b91b6cd518fb354e"
format_stamp: "Formatted at 2026-09-27 03:28:34 on dist-test-slave-j0hf"
server_key: "abfe75df613016505211f5a666b782f7"
server_key_iv: "8011178e648a9c8cc57f62d9451da462"
server_key_version: "encryptionkey@0"
I20260927 03:28:34.677666 8579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "be01327a73c94746b91b6cd518fb354e"
format_stamp: "Formatted at 2026-09-27 03:28:34 on dist-test-slave-j0hf"
server_key: "abfe75df613016505211f5a666b782f7"
server_key_iv: "8011178e648a9c8cc57f62d9451da462"
server_key_version: "encryptionkey@0"
I20260927 03:28:34.678926 8579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:28:34.679704 8594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:34.679873 8579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:34.679988 8579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "be01327a73c94746b91b6cd518fb354e"
format_stamp: "Formatted at 2026-09-27 03:28:34 on dist-test-slave-j0hf"
server_key: "abfe75df613016505211f5a666b782f7"
server_key_iv: "8011178e648a9c8cc57f62d9451da462"
server_key_version: "encryptionkey@0"
I20260927 03:28:34.680079 8579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:34.689972 8579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:34.691860 8579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:34.692013 8579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:34.695791 8579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36343
I20260927 03:28:34.695822 8646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36343 every 8 connection(s)
I20260927 03:28:34.696301 8579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:28:34.697011 8647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:34.699146 8647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e: Bootstrap starting.
I20260927 03:28:34.699756 8647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:34.699998 8647 log.cc:824] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:34.700634 8647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e: No bootstrap required, opened a new log
I20260927 03:28:34.702108 8647 raft_consensus.cc:348] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be01327a73c94746b91b6cd518fb354e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36343 } }
I20260927 03:28:34.702244 8647 raft_consensus.cc:374] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:34.702267 8647 raft_consensus.cc:729] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be01327a73c94746b91b6cd518fb354e, State: Initialized, Role: FOLLOWER
I20260927 03:28:34.702410 8647 consensus_queue.cc:260] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be01327a73c94746b91b6cd518fb354e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36343 } }
I20260927 03:28:34.702486 8647 raft_consensus.cc:388] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:28:34.702509 8647 raft_consensus.cc:482] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:28:34.702538 8647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:34.703153 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 8579
I20260927 03:28:34.703260 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:28:34.703517 392 external_mini_cluster.cc:1428] Setting key 81d45ff54b1a3c7a783bdf8c4c9da8dd
I20260927 03:28:34.703524 8647 raft_consensus.cc:504] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be01327a73c94746b91b6cd518fb354e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36343 } }
I20260927 03:28:34.703645 8647 leader_election.cc:302] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be01327a73c94746b91b6cd518fb354e; no voters:
I20260927 03:28:34.703840 8647 leader_election.cc:288] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:28:34.703891 8650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:34.704123 8650 raft_consensus.cc:686] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [term 1 LEADER]: Becoming Leader. State: Replica: be01327a73c94746b91b6cd518fb354e, State: Running, Role: LEADER
I20260927 03:28:34.704243 8647 sys_catalog.cc:564] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:28:34.704264 8650 consensus_queue.cc:237] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be01327a73c94746b91b6cd518fb354e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36343 } }
I20260927 03:28:34.704695 8652 sys_catalog.cc:455] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be01327a73c94746b91b6cd518fb354e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be01327a73c94746b91b6cd518fb354e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36343 } } }
I20260927 03:28:34.704844 8652 sys_catalog.cc:458] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:34.704881 8654 sys_catalog.cc:455] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [sys.catalog]: SysCatalogTable state changed. Reason: New leader be01327a73c94746b91b6cd518fb354e. Latest consensus state: current_term: 1 leader_uuid: "be01327a73c94746b91b6cd518fb354e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be01327a73c94746b91b6cd518fb354e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36343 } } }
I20260927 03:28:34.705065 8654 sys_catalog.cc:458] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:34.705329 8660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:28:34.705886 8660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:28:34.707494 8660 catalog_manager.cc:1347] Generated new cluster ID: c9f71de474364c828d031c794807213a
I20260927 03:28:34.707548 8660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:28:34 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:34.715977 8653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:34.704017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40248 (local address 127.0.98.62:36343)
0927 03:28:34.704352 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:34.704361 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:34.704396 (+ 35us) server_negotiation.cc:407] Connection header received
0927 03:28:34.704516 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:34.704525 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:34.704586 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:34.704726 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:34.705581 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:34.706962 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:34.708106 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:34.708724 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:34.708819 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:34.711666 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:34.711670 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:34.711671 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:34.711683 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:34.714315 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:34.714700 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:34.714703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:34.714703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:34.714812 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:34.715013 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:34.715014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:34.715014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:34.715344 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:34.715529 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:34.715602 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 03:28:34.715739 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":79,"threads_started":1}
I20260927 03:28:34.720881 8660 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:28:34.721432 8660 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:28:34.726967 8660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e: Generated new TSK 0
I20260927 03:28:34.727139 8660 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:28:34.752449 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36343
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:34.833487 8673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:34.833686 8673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:34.833709 8673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:34.834805 8673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:34.834839 8673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:34.834865 8673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:28:34.836362 8673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:34.836599 8673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:34.836786 8673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:34.841638 8678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:34.841638 8681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:34.841711 8679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:34.841988 8673 server_base.cc:1034] running on GCE node
I20260927 03:28:34.842218 8673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:34.842450 8673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:34.843606 8673 hybrid_clock.cc:648] HybridClock initialized: now 1790479714843595 us; error 27 us; skew 500 ppm
Sep 27 03:28:34 dist-test-slave-j0hf krb5kdc[8566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:34.846040 8673 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:28:34.846491 8673 webserver.cc:460] Webserver started at http://127.0.98.1:38329/ using document root <none> and password file <none>
I20260927 03:28:34.846726 8673 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:34.846801 8673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:34.846956 8673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:34.848064 8673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-0/data/instance:
uuid: "56e5a83cabfb4300b819b98b10b8d428"
format_stamp: "Formatted at 2026-09-27 03:28:34 on dist-test-slave-j0hf"
server_key: "d397aaa419fd9cabfbb269031917ecc8"
server_key_iv: "5d43b9c5085ff01fdb6f9e6ed635bc45"
server_key_version: "encryptionkey@0"
I20260927 03:28:34.848430 8673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-0/wal/instance:
uuid: "56e5a83cabfb4300b819b98b10b8d428"
format_stamp: "Formatted at 2026-09-27 03:28:34 on dist-test-slave-j0hf"
server_key: "d397aaa419fd9cabfbb269031917ecc8"
server_key_iv: "5d43b9c5085ff01fdb6f9e6ed635bc45"
server_key_version: "encryptionkey@0"
I20260927 03:28:34.851768 8673 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 03:28:34.852547 8688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:34.852706 8673 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:34.852780 8673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "56e5a83cabfb4300b819b98b10b8d428"
format_stamp: "Formatted at 2026-09-27 03:28:34 on dist-test-slave-j0hf"
server_key: "d397aaa419fd9cabfbb269031917ecc8"
server_key_iv: "5d43b9c5085ff01fdb6f9e6ed635bc45"
server_key_version: "encryptionkey@0"
I20260927 03:28:34.852835 8673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:34.865288 8673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:34.867189 8673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:34.867345 8673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:34.867581 8673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:34.867877 8673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:34.867929 8673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:34.867972 8673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:34.868000 8673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:34.873765 8673 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37065
I20260927 03:28:34.873800 8801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37065 every 8 connection(s)
I20260927 03:28:34.874156 8673 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:28:34.877151 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 8673
I20260927 03:28:34.877257 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:28:34.877457 392 external_mini_cluster.cc:1428] Setting key f9bd808e33d7b681d1984329333dc6e2
Sep 27 03:28:34 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:34.883909 8653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:34.874925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39187 (local address 127.0.98.62:36343)
0927 03:28:34.874975 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:34.874977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:34.875114 (+ 137us) server_negotiation.cc:407] Connection header received
0927 03:28:34.876134 (+ 1020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:34.876136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:34.876147 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:34.876219 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:34.876802 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:34.877393 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:34.878443 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:34.878657 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:34.878797 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:34.880896 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:34.880898 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:34.880898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:34.880906 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:34.882459 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:34.882893 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:34.882894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:34.882895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:34.882956 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:34.883261 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:34.883262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:34.883262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:34.883377 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:34.883478 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:34.883743 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 03:28:34.883821 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:28:34.884130 8804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:34.874834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.1:39187)
0927 03:28:34.874985 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:34.875025 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:34.876047 (+ 1022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:34.876292 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:34.876296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:34.876542 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:34.876723 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:34.876727 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:34.877480 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:34.877482 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:34.878350 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:34.878353 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:34.878806 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:34.878807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:34.878932 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:34.879482 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:34.879490 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:34.880798 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:34.882549 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:34.882556 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:34.882558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:34.882809 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:34.883023 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:34.883024 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:34.883024 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:34.883148 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:34.883461 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:34.883467 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:34.883609 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 03:28:34.883723 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 03:28:34.883875 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
I20260927 03:28:34.884846 8802 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36343
I20260927 03:28:34.884948 8802 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:34.885128 8802 heartbeater.cc:511] Master 127.0.98.62:36343 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 03:28:34.885674 8611 ts_manager.cc:194] Registered new tserver with Master: 56e5a83cabfb4300b819b98b10b8d428 (127.0.98.1:37065)
I20260927 03:28:34.886372 8611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39187
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:28:34.902470 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36343
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:34.983928 8807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:34.984139 8807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:34.984163 8807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:34.985291 8807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:34.985329 8807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:34.985355 8807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:28:34.986881 8807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:34.987073 8807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:34.987259 8807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:34.992122 8812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:34.992210 8813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:34.992236 8815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:34.992385 8807 server_base.cc:1034] running on GCE node
I20260927 03:28:34.992602 8807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:34.992883 8807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:34.994050 8807 hybrid_clock.cc:648] HybridClock initialized: now 1790479714994031 us; error 35 us; skew 500 ppm
Sep 27 03:28:34 dist-test-slave-j0hf krb5kdc[8566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:34.996794 8807 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:28:34.997270 8807 webserver.cc:460] Webserver started at http://127.0.98.2:44363/ using document root <none> and password file <none>
I20260927 03:28:34.997527 8807 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:34.997572 8807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:34.997712 8807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:34.998796 8807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-1/data/instance:
uuid: "dba575a644014d429efd35f4565eeca8"
format_stamp: "Formatted at 2026-09-27 03:28:34 on dist-test-slave-j0hf"
server_key: "d929520e0d3cfec43cb14094098a758a"
server_key_iv: "820d5635b747da1a656db20a6bae8ad5"
server_key_version: "encryptionkey@0"
I20260927 03:28:34.999156 8807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-1/wal/instance:
uuid: "dba575a644014d429efd35f4565eeca8"
format_stamp: "Formatted at 2026-09-27 03:28:34 on dist-test-slave-j0hf"
server_key: "d929520e0d3cfec43cb14094098a758a"
server_key_iv: "820d5635b747da1a656db20a6bae8ad5"
server_key_version: "encryptionkey@0"
I20260927 03:28:35.000573 8807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:28:35.001380 8822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:35.001593 8807 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:28:35.001664 8807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "dba575a644014d429efd35f4565eeca8"
format_stamp: "Formatted at 2026-09-27 03:28:34 on dist-test-slave-j0hf"
server_key: "d929520e0d3cfec43cb14094098a758a"
server_key_iv: "820d5635b747da1a656db20a6bae8ad5"
server_key_version: "encryptionkey@0"
I20260927 03:28:35.001748 8807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:35.022836 8807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:35.025385 8807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:35.025534 8807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:35.025803 8807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:35.026119 8807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:35.026194 8807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:35.026242 8807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:35.026288 8807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:35.032094 8807 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35491
I20260927 03:28:35.032140 8935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35491 every 8 connection(s)
I20260927 03:28:35.032475 8807 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:28:35.037465 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 8807
I20260927 03:28:35.037596 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:28:35.037803 392 external_mini_cluster.cc:1428] Setting key f30378242716d4ee169b6abe23a05fa0
Sep 27 03:28:35 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:35.043571 8653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:35.033299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49539 (local address 127.0.98.62:36343)
0927 03:28:35.033414 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:35.033416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:35.033602 (+ 186us) server_negotiation.cc:407] Connection header received
0927 03:28:35.034709 (+ 1107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:35.034711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:35.034726 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:35.034796 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:35.035536 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.036044 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.037164 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.037366 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.037461 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:35.040087 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:35.040089 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:35.040089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:35.040099 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:35.041774 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:35.042395 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:35.042396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:35.042396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:35.042502 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:35.042817 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:35.042818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:35.042818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:35.042933 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:35.042998 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:35.043373 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 03:28:35.043495 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 03:28:35.043896 8938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:35.033192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.2:49539)
0927 03:28:35.033444 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:35.033504 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:35.034551 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:35.034877 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:35.034881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:35.035174 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:35.035373 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:35.035379 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.036166 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.036167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:35.037041 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:35.037043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.037467 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.037468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:35.037644 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:35.038286 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:35.038295 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:35.039982 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:35.041885 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:35.041892 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:35.041894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:35.042304 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:35.042580 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:35.042581 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:35.042581 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:35.042734 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:35.043015 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:35.043022 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:35.043217 (+ 195us) client_negotiation.cc:769] Sending connection context
0927 03:28:35.043357 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 03:28:35.043582 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":106,"threads_started":1}
I20260927 03:28:35.044445 8936 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36343
I20260927 03:28:35.044549 8936 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:35.044744 8936 heartbeater.cc:511] Master 127.0.98.62:36343 requested a full tablet report, sending...
I20260927 03:28:35.045149 8611 ts_manager.cc:194] Registered new tserver with Master: dba575a644014d429efd35f4565eeca8 (127.0.98.2:35491)
I20260927 03:28:35.045534 8611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49539
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:28:35.064229 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36343
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:35.144927 8941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:35.145124 8941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:35.145147 8941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:35.146343 8941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:35.146384 8941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:35.146411 8941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:28:35.147943 8941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:35.148186 8941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:35.148423 8941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:35.153616 8949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:35.153582 8946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:35.153798 8941 server_base.cc:1034] running on GCE node
W20260927 03:28:35.153618 8947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:35.154099 8941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:35.154352 8941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:35.155526 8941 hybrid_clock.cc:648] HybridClock initialized: now 1790479715155529 us; error 50 us; skew 500 ppm
Sep 27 03:28:35 dist-test-slave-j0hf krb5kdc[8566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:35.158185 8941 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:28:35.158701 8941 webserver.cc:460] Webserver started at http://127.0.98.3:44213/ using document root <none> and password file <none>
I20260927 03:28:35.158965 8941 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:35.159044 8941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:35.159214 8941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:35.160352 8941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-2/data/instance:
uuid: "5a3a240e4c024415b4653104863ffc07"
format_stamp: "Formatted at 2026-09-27 03:28:35 on dist-test-slave-j0hf"
server_key: "b11d4f06ca07c1ac81e0f9dac78c3f1e"
server_key_iv: "84ab2f0b110ec40da3e7de18722ae8b3"
server_key_version: "encryptionkey@0"
I20260927 03:28:35.160714 8941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5a3a240e4c024415b4653104863ffc07"
format_stamp: "Formatted at 2026-09-27 03:28:35 on dist-test-slave-j0hf"
server_key: "b11d4f06ca07c1ac81e0f9dac78c3f1e"
server_key_iv: "84ab2f0b110ec40da3e7de18722ae8b3"
server_key_version: "encryptionkey@0"
I20260927 03:28:35.162051 8941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:28:35.162820 8956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:35.163094 8941 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:35.163199 8941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "5a3a240e4c024415b4653104863ffc07"
format_stamp: "Formatted at 2026-09-27 03:28:35 on dist-test-slave-j0hf"
server_key: "b11d4f06ca07c1ac81e0f9dac78c3f1e"
server_key_iv: "84ab2f0b110ec40da3e7de18722ae8b3"
server_key_version: "encryptionkey@0"
I20260927 03:28:35.163281 8941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:35.173635 8941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:35.175652 8941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:35.175827 8941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:35.176054 8941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:35.176476 8941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:35.176555 8941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:35.176594 8941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:35.176625 8941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:35.182580 8941 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37869
I20260927 03:28:35.182617 9069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37869 every 8 connection(s)
I20260927 03:28:35.183081 8941 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:28:35.189095 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 8941
I20260927 03:28:35.189188 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:28:35.189401 392 external_mini_cluster.cc:1428] Setting key 9b37652ce02deb86abcad3f0eda61534
Sep 27 03:28:35 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:35.192724 8653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:35.183948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43475 (local address 127.0.98.62:36343)
0927 03:28:35.184000 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:35.184003 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:35.184256 (+ 253us) server_negotiation.cc:407] Connection header received
0927 03:28:35.185170 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:35.185171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:35.185181 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:35.185244 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:35.185834 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.186330 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.187294 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.187551 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.187603 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:35.190032 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:35.190033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:35.190034 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:35.190041 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:35.191381 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:35.191863 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:35.191864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:35.191864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:35.191916 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:35.192137 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:35.192138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:35.192138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:35.192277 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:35.192347 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:35.192588 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 03:28:35.192659 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:28:35.192947 9072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:35.183915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.3:43475)
0927 03:28:35.184097 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:35.184140 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:35.185073 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:35.185303 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:35.185306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:35.185560 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:35.185759 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:35.185763 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.186407 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.186408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:35.187220 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:35.187222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.187610 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.187611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:35.187740 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:35.188341 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:35.188349 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:35.189964 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:35.191492 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:35.191499 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:35.191501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:35.191800 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:35.191959 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:35.191960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:35.191960 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:35.192045 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:35.192353 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:35.192358 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:35.192472 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 03:28:35.192557 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 03:28:35.192707 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260927 03:28:35.193475 9070 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36343
I20260927 03:28:35.193557 9070 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:35.193702 9070 heartbeater.cc:511] Master 127.0.98.62:36343 requested a full tablet report, sending...
I20260927 03:28:35.194053 8611 ts_manager.cc:194] Registered new tserver with Master: 5a3a240e4c024415b4653104863ffc07 (127.0.98.3:37869)
I20260927 03:28:35.194437 8611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43475
I20260927 03:28:35.201273 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 03:28:35.267549 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 8673
I20260927 03:28:35.271931 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 8807
I20260927 03:28:35.276371 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 8941
I20260927 03:28:35.280711 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 8579
I20260927 03:28:35.286104 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36343
--webserver_interface=127.0.98.62
--webserver_port=37867
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:35.368606 9079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:35.368893 9079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:35.368935 9079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:35.370150 9079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:28:35.370231 9079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:35.370258 9079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:35.370285 9079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:28:35.370317 9079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:28:35.371887 9079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:35.372184 9079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:35.372426 9079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:35.377261 9085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:35.377274 9084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:35.377261 9087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:35.377951 9079 server_base.cc:1034] running on GCE node
I20260927 03:28:35.378202 9079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:35.378494 9079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:35.379660 9079 hybrid_clock.cc:648] HybridClock initialized: now 1790479715379717 us; error 99 us; skew 500 ppm
Sep 27 03:28:35 dist-test-slave-j0hf krb5kdc[8566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:35.382217 9079 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:28:35.382638 9079 webserver.cc:460] Webserver started at http://127.0.98.62:37867/ using document root <none> and password file <none>
I20260927 03:28:35.382877 9079 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:35.382961 9079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:35.384354 9079 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:35.385003 9094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:35.385164 9079 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:35.385269 9079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "be01327a73c94746b91b6cd518fb354e"
format_stamp: "Formatted at 2026-09-27 03:28:34 on dist-test-slave-j0hf"
server_key: "abfe75df613016505211f5a666b782f7"
server_key_iv: "8011178e648a9c8cc57f62d9451da462"
server_key_version: "encryptionkey@0"
I20260927 03:28:35.385618 9079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:35.409158 9079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:35.411644 9079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:35.411821 9079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:35.415776 9079 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36343
I20260927 03:28:35.415843 9146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36343 every 8 connection(s)
I20260927 03:28:35.416384 9079 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:28:35.417706 9147 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:28:35.418373 9147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e: Bootstrap starting.
I20260927 03:28:35.421310 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 9079
I20260927 03:28:35.421626 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37065
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36343
--webserver_port=38329
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:28:35.427289 9147 log.cc:824] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:35.429236 9147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:28:35.429587 9147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e: Bootstrap complete.
I20260927 03:28:35.432060 9147 raft_consensus.cc:348] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be01327a73c94746b91b6cd518fb354e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36343 } }
I20260927 03:28:35.432317 9147 raft_consensus.cc:729] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: be01327a73c94746b91b6cd518fb354e, State: Initialized, Role: FOLLOWER
I20260927 03:28:35.432497 9147 consensus_queue.cc:260] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be01327a73c94746b91b6cd518fb354e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36343 } }
I20260927 03:28:35.432574 9147 raft_consensus.cc:388] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:28:35.432626 9147 raft_consensus.cc:482] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:28:35.432689 9147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [term 1 FOLLOWER]: Advancing to term 2
I20260927 03:28:35.433775 9147 raft_consensus.cc:504] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be01327a73c94746b91b6cd518fb354e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36343 } }
I20260927 03:28:35.433898 9147 leader_election.cc:302] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be01327a73c94746b91b6cd518fb354e; no voters:
I20260927 03:28:35.434111 9147 leader_election.cc:288] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 03:28:35.434156 9151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [term 2 FOLLOWER]: Leader election won for term 2
I20260927 03:28:35.434279 9151 raft_consensus.cc:686] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [term 2 LEADER]: Becoming Leader. State: Replica: be01327a73c94746b91b6cd518fb354e, State: Running, Role: LEADER
I20260927 03:28:35.434371 9151 consensus_queue.cc:237] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be01327a73c94746b91b6cd518fb354e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36343 } }
I20260927 03:28:35.434401 9147 sys_catalog.cc:564] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:28:35.434655 9153 sys_catalog.cc:455] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "be01327a73c94746b91b6cd518fb354e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be01327a73c94746b91b6cd518fb354e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36343 } } }
I20260927 03:28:35.434803 9153 sys_catalog.cc:458] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:35.435134 9152 sys_catalog.cc:455] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [sys.catalog]: SysCatalogTable state changed. Reason: New leader be01327a73c94746b91b6cd518fb354e. Latest consensus state: current_term: 2 leader_uuid: "be01327a73c94746b91b6cd518fb354e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be01327a73c94746b91b6cd518fb354e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36343 } } }
I20260927 03:28:35.435245 9152 sys_catalog.cc:458] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:35.435384 9156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:28:35.435943 9156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:28:35.436081 9156 catalog_manager.cc:1259] Loaded cluster ID: c9f71de474364c828d031c794807213a
I20260927 03:28:35.436110 9156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:28:35.437340 9156 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:28:35.437520 9156 catalog_manager.cc:5879] T 00000000000000000000000000000000 P be01327a73c94746b91b6cd518fb354e: Loaded TSK: 0
I20260927 03:28:35.437743 9156 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 03:28:35.507692 9149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:35.507922 9149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:35.507961 9149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:35.509310 9149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:35.509382 9149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:35.509428 9149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:28:35.510895 9149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37065
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38329
--tserver_master_addrs=127.0.98.62:36343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:35.511169 9149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:35.511456 9149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:35.517426 9174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:35.517516 9176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:35.517575 9149 server_base.cc:1034] running on GCE node
W20260927 03:28:35.517426 9173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:35.517823 9149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:35.518092 9149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:35.519244 9149 hybrid_clock.cc:648] HybridClock initialized: now 1790479715519226 us; error 34 us; skew 500 ppm
Sep 27 03:28:35 dist-test-slave-j0hf krb5kdc[8566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479715, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:35.521965 9149 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 03:28:35.522506 9149 webserver.cc:460] Webserver started at http://127.0.98.1:38329/ using document root <none> and password file <none>
I20260927 03:28:35.522807 9149 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:35.522890 9149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:35.524684 9149 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:28:35.525354 9183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:35.525559 9149 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:28:35.525653 9149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "56e5a83cabfb4300b819b98b10b8d428"
format_stamp: "Formatted at 2026-09-27 03:28:34 on dist-test-slave-j0hf"
server_key: "d397aaa419fd9cabfbb269031917ecc8"
server_key_iv: "5d43b9c5085ff01fdb6f9e6ed635bc45"
server_key_version: "encryptionkey@0"
I20260927 03:28:35.526090 9149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:35.539503 9149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:35.542074 9149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:35.542235 9149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:35.542487 9149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:35.542840 9149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:35.542889 9149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:35.542919 9149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:35.542936 9149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:35.550222 9149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37065
I20260927 03:28:35.550297 9296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37065 every 8 connection(s)
I20260927 03:28:35.550689 9149 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:28:35.557332 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 9149
I20260927 03:28:35.557673 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35491
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36343
--webserver_port=44363
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 03:28:35 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:35 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:35.560043 9299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:35.551469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.1:32995)
0927 03:28:35.551668 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:35.551712 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:35.552207 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:35.552747 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:35.552751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:35.553361 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:35.553570 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:35.553575 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.554753 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.554754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:35.555651 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:35.555653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.556284 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.556285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:35.556427 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:35.557001 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:35.557009 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:35.559825 (+ 2816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":123,"threads_started":1}
W20260927 03:28:35.560127 9300 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:35.551551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32995 (local address 127.0.98.62:36343)
0927 03:28:35.552370 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:35.552376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:35.552403 (+ 27us) server_negotiation.cc:407] Connection header received
0927 03:28:35.552495 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:35.552499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:35.552538 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:35.552683 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:35.553711 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.554616 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.555758 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.556172 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.556310 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:35.559950 (+ 3640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:32995: BlockingRecv error: recv got EOF from 127.0.98.1:32995 (error 108)
Metrics: {"server-negotiator.queue_time_us":772,"thread_start_us":88,"threads_started":1}
W20260927 03:28:35.560283 9297 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36343 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36343: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 03:28:35 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:35 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:35.566268 9299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:35.560519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.1:50349)
0927 03:28:35.560562 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:35.560572 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:35.560609 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:35.560847 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:35.560848 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:35.561013 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:35.561072 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:35.561074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.562001 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.562003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:35.562906 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:35.562909 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.563470 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.563472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:35.563557 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:35.564068 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:35.564074 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:35.566198 (+ 2124us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 03:28:35.566437 9302 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:35.560567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50349 (local address 127.0.98.62:36343)
0927 03:28:35.560703 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:35.560705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:35.560714 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:28:35.560736 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:35.560736 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:35.560746 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:35.560827 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:35.561159 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.561911 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.563023 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.563372 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.563461 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:35.566328 (+ 2867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50349: BlockingRecv error: recv got EOF from 127.0.98.1:50349 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
Sep 27 03:28:35 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:35 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:35.572479 9299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:35.566722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.1:56531)
0927 03:28:35.566772 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:35.566783 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:35.566823 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:35.567225 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:35.567226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:35.567382 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:35.567448 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:35.567450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.568413 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.568415 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:35.569284 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:35.569287 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.569774 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.569775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:35.569834 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:35.570337 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:35.570342 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:35.572407 (+ 2065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 03:28:35.572585 9300 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:35.566780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56531 (local address 127.0.98.62:36343)
0927 03:28:35.566815 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:35.566817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:35.566826 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:28:35.566892 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:35.566893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:35.566904 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:35.567198 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:35.567528 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.568309 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.569386 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.569689 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.569758 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:35.572519 (+ 2761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56531: BlockingRecv error: recv got EOF from 127.0.98.1:56531 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 03:28:35.572860 9297 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 03:28:35.643510 9301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:35.643754 9301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:35.643781 9301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:35.645005 9301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:35.645056 9301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:35.645121 9301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:28:35.646616 9301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44363
--tserver_master_addrs=127.0.98.62:36343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:35.646885 9301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:35.647128 9301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:35.652034 9307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:35.652055 9310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:35.652165 9308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:35.652441 9301 server_base.cc:1034] running on GCE node
I20260927 03:28:35.652678 9301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:35.652913 9301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:35.654068 9301 hybrid_clock.cc:648] HybridClock initialized: now 1790479715654058 us; error 28 us; skew 500 ppm
Sep 27 03:28:35 dist-test-slave-j0hf krb5kdc[8566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479715, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:35.656579 9301 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 03:28:35.657006 9301 webserver.cc:460] Webserver started at http://127.0.98.2:44363/ using document root <none> and password file <none>
I20260927 03:28:35.657240 9301 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:35.657321 9301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:35.658650 9301 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:35.659216 9317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:35.659384 9301 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:35.659509 9301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "dba575a644014d429efd35f4565eeca8"
format_stamp: "Formatted at 2026-09-27 03:28:34 on dist-test-slave-j0hf"
server_key: "d929520e0d3cfec43cb14094098a758a"
server_key_iv: "820d5635b747da1a656db20a6bae8ad5"
server_key_version: "encryptionkey@0"
I20260927 03:28:35.659881 9301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:35.692967 9301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:35.695161 9301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:35.695340 9301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:35.695613 9301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:35.695931 9301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:35.695984 9301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:35.696019 9301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:35.696045 9301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:35.702016 9301 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35491
I20260927 03:28:35.702082 9430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35491 every 8 connection(s)
I20260927 03:28:35.702476 9301 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:28:35.703789 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 9301
I20260927 03:28:35.704100 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37869
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36343
--webserver_port=44213
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 03:28:35 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:35 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 03:28:35.711081 9300 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:35.703342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59777 (local address 127.0.98.62:36343)
0927 03:28:35.703457 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:35.703459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:35.703645 (+ 186us) server_negotiation.cc:407] Connection header received
0927 03:28:35.704239 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:35.704240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:35.704257 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:35.704316 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:35.705380 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.705935 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.707066 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.707272 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.707343 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:35.711030 (+ 3687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59777: BlockingRecv error: recv got EOF from 127.0.98.2:59777 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 03:28:35.711191 9433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:35.703320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.2:59777)
0927 03:28:35.703493 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:35.703538 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:35.704115 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:35.704400 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:35.704404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:35.704991 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:35.705233 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:35.705242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.706042 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.706043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:35.706975 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:35.706978 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.707364 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.707365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:35.707540 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:35.708115 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:35.708126 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:35.710936 (+ 2810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":110,"threads_started":1}
W20260927 03:28:35.711498 9431 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36343 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36343: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 03:28:35 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:35 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:35.716699 9433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:35.711767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.2:59461)
0927 03:28:35.711814 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:35.711825 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:35.711866 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:35.712006 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:35.712007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:35.712165 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:35.712224 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:35.712225 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.712902 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.712903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:35.713837 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:35.713840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.714177 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.714178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:35.714256 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:35.714671 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:35.714684 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:35.716612 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 03:28:35.716763 9300 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:35.711815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59461 (local address 127.0.98.62:36343)
0927 03:28:35.711868 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:35.711870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:35.711879 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:28:35.711925 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:35.711926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:35.711935 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:35.711984 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:35.712303 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.712818 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.713930 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.714105 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.714170 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:35.716687 (+ 2517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59461: BlockingRecv error: recv got EOF from 127.0.98.2:59461 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 27 03:28:35 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:35 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:35.721351 9433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:35.717026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.2:42665)
0927 03:28:35.717081 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:35.717092 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:35.717134 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:35.717299 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:35.717299 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:35.717422 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:35.717448 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:35.717449 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.718044 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.718045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:35.718550 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:35.718551 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.718937 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.718938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:35.718988 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:35.719507 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:35.719515 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:35.721277 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 03:28:35.721460 9300 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:35.717102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42665 (local address 127.0.98.62:36343)
0927 03:28:35.717139 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:35.717141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:35.717149 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:28:35.717192 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:35.717193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:35.717205 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:35.717293 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:35.717522 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.717966 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.718682 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.718854 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.718927 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:35.721370 (+ 2443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42665: BlockingRecv error: recv got EOF from 127.0.98.2:42665 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 03:28:35.721478 9431 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 03:28:35.789834 9434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:35.790037 9434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:35.790061 9434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:35.791240 9434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:35.791285 9434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:35.791312 9434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:28:35.792785 9434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44213
--tserver_master_addrs=127.0.98.62:36343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:35.793017 9434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:35.793251 9434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:35.798553 9442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:35.798508 9440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:35.798502 9439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:35.798727 9434 server_base.cc:1034] running on GCE node
I20260927 03:28:35.798964 9434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:35.799219 9434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:35.800405 9434 hybrid_clock.cc:648] HybridClock initialized: now 1790479715800383 us; error 53 us; skew 500 ppm
Sep 27 03:28:35 dist-test-slave-j0hf krb5kdc[8566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479715, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:35.802940 9434 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 03:28:35.803467 9434 webserver.cc:460] Webserver started at http://127.0.98.3:44213/ using document root <none> and password file <none>
I20260927 03:28:35.803681 9434 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:35.803745 9434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:35.805122 9434 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:35.805745 9449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:35.805958 9434 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:35.806061 9434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "5a3a240e4c024415b4653104863ffc07"
format_stamp: "Formatted at 2026-09-27 03:28:35 on dist-test-slave-j0hf"
server_key: "b11d4f06ca07c1ac81e0f9dac78c3f1e"
server_key_iv: "84ab2f0b110ec40da3e7de18722ae8b3"
server_key_version: "encryptionkey@0"
I20260927 03:28:35.806437 9434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:35.828359 9434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:35.830905 9434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:35.831097 9434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:35.831450 9434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:35.831833 9434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:35.831897 9434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:35.831941 9434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:35.831972 9434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:35.838714 9434 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37869
I20260927 03:28:35.838750 9562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37869 every 8 connection(s)
I20260927 03:28:35.839143 9434 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:28:35.840428 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 9434
Sep 27 03:28:35 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:35 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 03:28:35.847388 9300 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:35.839999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45679 (local address 127.0.98.62:36343)
0927 03:28:35.840075 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:35.840078 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:35.840260 (+ 182us) server_negotiation.cc:407] Connection header received
0927 03:28:35.840960 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:35.840962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:35.840976 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:35.841038 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:35.842019 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.842514 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.843538 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.843726 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.843832 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:35.847331 (+ 3499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45679: BlockingRecv error: recv got EOF from 127.0.98.3:45679 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 03:28:35.847532 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:35.839973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.3:45679)
0927 03:28:35.840120 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:35.840168 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:35.840838 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:35.841120 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:35.841123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:35.841693 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:35.841908 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:35.841912 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.842611 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.842612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:35.843442 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:35.843469 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.843852 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.843853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:35.843979 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:35.844618 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:35.844626 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:35.847249 (+ 2623us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
W20260927 03:28:35.847826 9563 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36343 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36343: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260927 03:28:35.848850 9302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:35.840987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40262 (local address 127.0.98.62:36343)
0927 03:28:35.841035 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:35.841037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:35.841227 (+ 190us) server_negotiation.cc:407] Connection header received
0927 03:28:35.841309 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:35.841310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:35.841321 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:35.841383 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:35.842079 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.842608 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.843491 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.843716 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.843838 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:35.845301 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:35.845306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:35.845306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:35.845318 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:35.847107 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:35.847566 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:35.847570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:35.847570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:35.847706 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:35.847993 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:35.847994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:35.847995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:35.848301 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:35.848405 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:35.848596 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 03:28:35.848776 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 27 03:28:35 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:35 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:35.852861 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:35.848060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.3:59817)
0927 03:28:35.848434 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:35.848443 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:35.848528 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:35.848700 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:35.848701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:35.848879 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:35.848954 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:35.848956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.849868 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.849869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:35.850371 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:35.850373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.850698 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.850699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:35.850743 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:35.851172 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:35.851177 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:35.852818 (+ 1641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358}
W20260927 03:28:35.852965 9300 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:35.848110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59817 (local address 127.0.98.62:36343)
0927 03:28:35.848168 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:35.848169 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:35.848501 (+ 332us) server_negotiation.cc:407] Connection header received
0927 03:28:35.848597 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:35.848598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:35.848609 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:35.848637 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:35.849038 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.849797 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.850446 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.850634 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.850689 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:35.852894 (+ 2205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59817: BlockingRecv error: recv got EOF from 127.0.98.3:59817 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 03:28:35 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:35 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:35.856938 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:35.853136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.3:52023)
0927 03:28:35.853185 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:35.853196 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:35.853231 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:35.853363 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:35.853364 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:35.853477 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:35.853506 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:35.853507 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.854098 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.854099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:35.854565 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:35.854566 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.854902 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.854902 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:35.854936 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:35.855335 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:35.855339 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:35.856896 (+ 1557us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 03:28:35.857054 9300 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:35.853186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52023 (local address 127.0.98.62:36343)
0927 03:28:35.853240 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:35.853242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:35.853281 (+ 39us) server_negotiation.cc:407] Connection header received
0927 03:28:35.853298 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:35.853299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:35.853306 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:35.853348 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:35.853568 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.854036 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.854652 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:35.854834 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:35.854896 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:35.856980 (+ 2084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52023: BlockingRecv error: recv got EOF from 127.0.98.3:52023 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 03:28:35.857118 9563 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 03:28:36 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:36 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:36.577921 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:36.573299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.1:50011)
0927 03:28:36.573465 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:36.573476 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:36.573538 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:36.573672 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:36.573673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:36.573861 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:36.573909 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:36.573911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:36.574564 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:36.574564 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:36.575217 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:36.575220 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:36.575610 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:36.575611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:36.575650 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:36.576102 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:36.576106 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:36.577829 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":106,"threads_started":1}
W20260927 03:28:36.578078 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:36.573409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50011 (local address 127.0.98.62:36343)
0927 03:28:36.573515 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:36.573517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:36.573533 (+ 16us) server_negotiation.cc:407] Connection header received
0927 03:28:36.573585 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:36.573586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:36.573596 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:36.573657 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:36.573983 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:36.574497 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:36.575310 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:36.575542 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:36.575603 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:36.577987 (+ 2384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50011: BlockingRecv error: recv got EOF from 127.0.98.1:50011 (error 108)
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":50,"threads_started":1}
Sep 27 03:28:36 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:36 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:36.726367 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:36.721923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.2:37255)
0927 03:28:36.722073 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:36.722083 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:36.722140 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:36.722272 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:36.722273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:36.722432 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:36.722479 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:36.722480 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:36.723157 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:36.723157 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:36.723762 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:36.723765 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:36.724103 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:36.724103 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:36.724142 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:36.724594 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:36.724598 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:36.726295 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
W20260927 03:28:36.726496 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:36.721998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37255 (local address 127.0.98.62:36343)
0927 03:28:36.722081 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:36.722083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:36.722140 (+ 57us) server_negotiation.cc:407] Connection header received
0927 03:28:36.722183 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:36.722184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:36.722198 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:36.722249 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:36.722547 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:36.723084 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:36.723833 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:36.724018 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:36.724076 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:36.726393 (+ 2317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37255: BlockingRecv error: recv got EOF from 127.0.98.2:37255 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 03:28:36 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:36 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:36.862895 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:36.857535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.3:60703)
0927 03:28:36.857739 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:36.857752 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:36.857833 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:36.857975 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:36.857976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:36.858162 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:36.858218 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:36.858221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:36.858864 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:36.858866 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:36.859513 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:36.859516 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:36.859875 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:36.859877 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:36.859937 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:36.860448 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:36.860453 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:36.862815 (+ 2362us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":134,"threads_started":1}
W20260927 03:28:36.862963 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:36.857624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60703 (local address 127.0.98.62:36343)
0927 03:28:36.857697 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:36.857699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:36.857832 (+ 133us) server_negotiation.cc:407] Connection header received
0927 03:28:36.857885 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:36.857886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:36.857896 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:36.857949 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:36.858290 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:36.858777 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:36.859599 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:36.859790 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:36.859844 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:36.862915 (+ 3071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60703: BlockingRecv error: recv got EOF from 127.0.98.3:60703 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 03:28:37 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:37 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:37.583642 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:37.578435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.1:54189)
0927 03:28:37.578600 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:37.578618 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:37.578701 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:37.578910 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:37.578911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:37.579072 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:37.579121 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:37.579122 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:37.579933 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:37.579934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:37.580464 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:37.580465 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:37.580873 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:37.580873 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:37.580915 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:37.581471 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:37.581476 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:37.583581 (+ 2105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":100,"threads_started":1}
W20260927 03:28:37.583820 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:37.578566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54189 (local address 127.0.98.62:36343)
0927 03:28:37.578700 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:37.578702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:37.578751 (+ 49us) server_negotiation.cc:407] Connection header received
0927 03:28:37.578784 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:37.578784 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:37.578796 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:37.578856 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:37.579214 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:37.579825 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:37.580557 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:37.580799 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:37.580866 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:37.583734 (+ 2868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54189: BlockingRecv error: recv got EOF from 127.0.98.1:54189 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
Sep 27 03:28:37 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:37 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:37.731621 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:37.726843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.2:54659)
0927 03:28:37.727016 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:37.727028 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:37.727082 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:37.727273 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:37.727274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:37.727461 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:37.727512 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:37.727514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:37.728167 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:37.728168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:37.728684 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:37.728686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:37.729061 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:37.729062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:37.729110 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:37.729660 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:37.729664 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:37.731563 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":99,"threads_started":1}
W20260927 03:28:37.731761 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:37.727007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54659 (local address 127.0.98.62:36343)
0927 03:28:37.727067 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:37.727069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:37.727082 (+ 13us) server_negotiation.cc:407] Connection header received
0927 03:28:37.727150 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:37.727150 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:37.727160 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:37.727217 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:37.727585 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:37.728090 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:37.728778 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:37.728983 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:37.729044 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:37.731688 (+ 2644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54659: BlockingRecv error: recv got EOF from 127.0.98.2:54659 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 27 03:28:37 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:37 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:37.868217 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:37.863471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.3:56511)
0927 03:28:37.863626 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:37.863637 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:37.863716 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:37.863889 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:37.863890 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:37.864066 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:37.864113 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:37.864114 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:37.864782 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:37.864783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:37.865310 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:37.865312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:37.865707 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:37.865708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:37.865773 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:37.866274 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:37.866279 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:37.868116 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
W20260927 03:28:37.868320 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:37.863624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56511 (local address 127.0.98.62:36343)
0927 03:28:37.863696 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:37.863698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:37.863708 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:28:37.863790 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:37.863791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:37.863811 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:37.863882 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:37.864195 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:37.864688 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:37.865418 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:37.865619 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:37.865705 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:37.868194 (+ 2489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56511: BlockingRecv error: recv got EOF from 127.0.98.3:56511 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 03:28:38 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:38 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:38.589035 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:38.584187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.1:50447)
0927 03:28:38.584339 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:38.584354 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:38.584424 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:38.584556 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:38.584557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:38.584725 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:38.584774 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:38.584776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:38.585429 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:38.585430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:38.585958 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:38.585959 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:38.586343 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:38.586344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:38.586387 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:38.586940 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:38.586944 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:38.588930 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":75,"threads_started":1}
W20260927 03:28:38.589088 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:38.584259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50447 (local address 127.0.98.62:36343)
0927 03:28:38.584411 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:38.584412 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:38.584421 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:28:38.584470 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:38.584470 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:38.584481 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:38.584540 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:38.584842 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:38.585355 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:38.586046 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:38.586271 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:38.586334 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:38.589015 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50447: BlockingRecv error: recv got EOF from 127.0.98.1:50447 (error 108)
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":96,"threads_started":1}
Sep 27 03:28:38 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:38 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:38.737756 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:38.732171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.2:41715)
0927 03:28:38.732331 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:38.732346 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:38.732438 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:38.732641 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:38.732642 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:38.732894 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:38.732975 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:38.732978 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:38.733832 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:38.733833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:38.734485 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:38.734487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:38.734971 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:38.734972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:38.735063 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:38.735634 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:38.735641 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:38.737672 (+ 2031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":104,"threads_started":1}
W20260927 03:28:38.737895 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:38.732266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41715 (local address 127.0.98.62:36343)
0927 03:28:38.732351 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:38.732353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:38.732431 (+ 78us) server_negotiation.cc:407] Connection header received
0927 03:28:38.732517 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:38.732536 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:38.732552 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:38.732630 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:38.733083 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:38.733740 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:38.734603 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:38.734862 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:38.734969 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:38.737801 (+ 2832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41715: BlockingRecv error: recv got EOF from 127.0.98.2:41715 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 03:28:38 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:38 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 03:28:38.873809 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:38.868849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42105 (local address 127.0.98.62:36343)
0927 03:28:38.868922 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:38.868923 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:38.868953 (+ 30us) server_negotiation.cc:407] Connection header received
0927 03:28:38.869014 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:38.869015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:38.869028 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:38.869077 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:38.869375 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:38.869885 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:38.870712 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:38.870897 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:38.870953 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:38.873751 (+ 2798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42105: BlockingRecv error: recv got EOF from 127.0.98.3:42105 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 03:28:38.873803 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:38.868724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.3:42105)
0927 03:28:38.868878 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:38.868891 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:38.868962 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:38.869096 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:38.869097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:38.869261 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:38.869309 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:38.869311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:38.869981 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:38.869982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:38.870593 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:38.870595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:38.870973 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:38.870974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:38.871021 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:38.871577 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:38.871582 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:38.873664 (+ 2082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":57,"threads_started":1}
Sep 27 03:28:39 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:39 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:39.594676 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:39.589600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.1:48487)
0927 03:28:39.589792 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:39.589804 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:39.589879 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:39.590040 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:39.590041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:39.590219 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:39.590270 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:39.590272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:39.591034 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:39.591035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:39.591676 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:39.591679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:39.592091 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:39.592092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:39.592154 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:39.592700 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:39.592705 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:39.594578 (+ 1873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":126,"threads_started":1}
W20260927 03:28:39.594727 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:39.589683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48487 (local address 127.0.98.62:36343)
0927 03:28:39.589823 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:39.589824 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:39.589862 (+ 38us) server_negotiation.cc:407] Connection header received
0927 03:28:39.589935 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:39.589935 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:39.589946 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:39.590017 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:39.590369 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:39.590932 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:39.591773 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:39.591995 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:39.592092 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:39.594662 (+ 2570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48487: BlockingRecv error: recv got EOF from 127.0.98.1:48487 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
Sep 27 03:28:39 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:39 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:39.747006 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:39.738326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.2:54737)
0927 03:28:39.738528 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:39.738540 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:39.738643 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:39.738806 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:39.738807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:39.738994 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:39.739040 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:39.739042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:39.739775 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:39.739776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:39.740314 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:39.740316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:39.740704 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:39.740706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:39.740768 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:39.745148 (+ 4380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:39.745152 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:39.746947 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":97,"threads_started":1}
W20260927 03:28:39.747220 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:39.738428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54737 (local address 127.0.98.62:36343)
0927 03:28:39.738511 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:39.738513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:39.738622 (+ 109us) server_negotiation.cc:407] Connection header received
0927 03:28:39.738706 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:39.738707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:39.738721 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:39.738798 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:39.739124 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:39.739690 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:39.740404 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:39.740608 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:39.740704 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:39.747137 (+ 6433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54737: BlockingRecv error: recv got EOF from 127.0.98.2:54737 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 03:28:39 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:39 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:39.879050 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:39.874334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.3:37155)
0927 03:28:39.874472 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:39.874483 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:39.874573 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:39.874773 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:39.874775 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:39.874953 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:39.874999 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:39.875000 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:39.875745 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:39.875747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:39.876286 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:39.876288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:39.876654 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:39.876655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:39.876714 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:39.877203 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:39.877207 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:39.878954 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":69,"threads_started":1}
W20260927 03:28:39.879104 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:39.874373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37155 (local address 127.0.98.62:36343)
0927 03:28:39.874437 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:39.874440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:39.874562 (+ 122us) server_negotiation.cc:407] Connection header received
0927 03:28:39.874631 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:39.874631 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:39.874647 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:39.874748 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:39.875081 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:39.875641 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:39.876376 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:39.876565 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:39.876653 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:39.879040 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37155: BlockingRecv error: recv got EOF from 127.0.98.3:37155 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 03:28:40 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:40 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:40.600160 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:40.595300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.1:42399)
0927 03:28:40.595416 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:40.595471 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:40.595534 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:40.595735 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:40.595736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:40.595971 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:40.596036 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:40.596038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:40.596674 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:40.596675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:40.597209 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:40.597211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:40.597558 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:40.597559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:40.597595 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:40.598021 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:40.598025 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:40.600086 (+ 2061us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":68,"threads_started":1}
W20260927 03:28:40.600261 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:40.595390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42399 (local address 127.0.98.62:36343)
0927 03:28:40.595566 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:40.595568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:40.595578 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:28:40.595612 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:40.595612 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:40.595622 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:40.595704 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:40.596094 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:40.596596 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:40.597301 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:40.597491 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:40.597547 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:40.600177 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42399: BlockingRecv error: recv got EOF from 127.0.98.1:42399 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":128,"threads_started":1}
Sep 27 03:28:40 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:40 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:40.752763 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:40.747589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.2:37049)
0927 03:28:40.747778 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:40.747793 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:40.747899 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:40.748089 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:40.748089 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:40.748283 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:40.748335 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:40.748337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:40.749031 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:40.749033 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:40.749596 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:40.749598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:40.750022 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:40.750023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:40.750090 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:40.750552 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:40.750557 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:40.752692 (+ 2135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":76,"threads_started":1}
W20260927 03:28:40.752838 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:40.747807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37049 (local address 127.0.98.62:36343)
0927 03:28:40.747884 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:40.747886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:40.747899 (+ 13us) server_negotiation.cc:407] Connection header received
0927 03:28:40.747973 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:40.747974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:40.747988 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:40.748071 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:40.748429 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:40.748937 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:40.749706 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:40.749908 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:40.750008 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:40.752792 (+ 2784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37049: BlockingRecv error: recv got EOF from 127.0.98.2:37049 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 03:28:40 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:40 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:40.884894 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:40.879789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.3:54717)
0927 03:28:40.880014 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:40.880027 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:40.880144 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:40.880310 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:40.880311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:40.880503 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:40.880552 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:40.880554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:40.881240 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:40.881241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:40.881816 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:40.881818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:40.882187 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:40.882188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:40.882245 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:40.882748 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:40.882752 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:40.884785 (+ 2033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":120,"threads_started":1}
W20260927 03:28:40.884932 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:40.879915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54717 (local address 127.0.98.62:36343)
0927 03:28:40.879998 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:40.880000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:40.880115 (+ 115us) server_negotiation.cc:407] Connection header received
0927 03:28:40.880216 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:40.880217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:40.880236 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:40.880302 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:40.880648 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:40.881148 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:40.881912 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:40.882105 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:40.882178 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:40.884867 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54717: BlockingRecv error: recv got EOF from 127.0.98.3:54717 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 03:28:41 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:41 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:41.605183 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:41.600688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.1:37081)
0927 03:28:41.600900 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:41.600913 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:41.600996 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:41.601140 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:41.601140 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:41.601307 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:41.601355 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:41.601357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:41.601979 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:41.601980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:41.602504 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:41.602505 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:41.602825 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:41.602826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:41.602862 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:41.603309 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:41.603314 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:41.605116 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":115,"threads_started":1}
W20260927 03:28:41.605312 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:41.600817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37081 (local address 127.0.98.62:36343)
0927 03:28:41.600947 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:41.600949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:41.600985 (+ 36us) server_negotiation.cc:407] Connection header received
0927 03:28:41.601041 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:41.601041 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:41.601053 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:41.601120 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:41.601420 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:41.601913 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:41.602569 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:41.602763 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:41.602813 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:41.605220 (+ 2407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37081: BlockingRecv error: recv got EOF from 127.0.98.1:37081 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":54,"threads_started":1}
Sep 27 03:28:41 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:41 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:41.757846 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:41.753267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.2:39445)
0927 03:28:41.753413 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:41.753423 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:41.753469 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:41.753594 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:41.753595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:41.753753 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:41.753799 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:41.753800 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:41.754441 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:41.754442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:41.754996 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:41.754997 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:41.755384 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:41.755385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:41.755469 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:41.755938 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:41.755942 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:41.757743 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
W20260927 03:28:41.757899 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:41.753391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39445 (local address 127.0.98.62:36343)
0927 03:28:41.753449 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:41.753450 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:41.753463 (+ 13us) server_negotiation.cc:407] Connection header received
0927 03:28:41.753516 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:41.753517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:41.753527 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:41.753584 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:41.753862 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:41.754370 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:41.755089 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:41.755316 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:41.755377 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:41.757829 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39445: BlockingRecv error: recv got EOF from 127.0.98.2:39445 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 27 03:28:41 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:41 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:41.890154 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:41.885440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.3:51979)
0927 03:28:41.885607 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:41.885617 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:41.885670 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:41.885793 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:41.885793 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:41.885955 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:41.886002 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:41.886003 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:41.886660 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:41.886661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:41.887171 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:41.887172 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:41.887571 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:41.887573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:41.887627 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:41.888087 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:41.888091 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:41.890048 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":107,"threads_started":1}
W20260927 03:28:41.890195 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:41.885539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51979 (local address 127.0.98.62:36343)
0927 03:28:41.885601 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:41.885603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:41.885674 (+ 71us) server_negotiation.cc:407] Connection header received
0927 03:28:41.885712 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:41.885713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:41.885723 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:41.885776 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:41.886096 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:41.886587 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:41.887251 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:41.887446 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:41.887493 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:41.890130 (+ 2637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51979: BlockingRecv error: recv got EOF from 127.0.98.3:51979 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 03:28:42 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:42 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:42.610574 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:42.605719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.1:39599)
0927 03:28:42.605859 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:42.605869 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:42.605920 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:42.606123 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:42.606125 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:42.606296 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:42.606345 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:42.606346 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:42.607023 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:42.607024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:42.607687 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:42.607690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:42.608046 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:42.608046 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:42.608095 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:42.608546 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:42.608551 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:42.610516 (+ 1965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
W20260927 03:28:42.610718 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:42.605829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39599 (local address 127.0.98.62:36343)
0927 03:28:42.605957 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:42.605961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:42.605974 (+ 13us) server_negotiation.cc:407] Connection header received
0927 03:28:42.605996 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:42.605997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:42.606007 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:42.606071 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:42.606418 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:42.606921 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:42.607770 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:42.607984 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:42.608036 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:42.610631 (+ 2595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39599: BlockingRecv error: recv got EOF from 127.0.98.1:39599 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
Sep 27 03:28:42 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:42 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:42.763285 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:42.758335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.2:51671)
0927 03:28:42.758471 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:42.758480 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:42.758521 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:42.758657 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:42.758658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:42.758840 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:42.758887 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:42.758889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:42.759608 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:42.759610 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:42.760187 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:42.760190 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:42.760559 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:42.760560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:42.760610 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:42.761252 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:42.761256 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:42.763181 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
W20260927 03:28:42.763339 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:42.758460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51671 (local address 127.0.98.62:36343)
0927 03:28:42.758536 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:42.758537 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:42.758546 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:28:42.758570 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:42.758570 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:42.758581 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:42.758636 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:42.758957 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:42.759516 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:42.760286 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:42.760480 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:42.760539 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:42.763275 (+ 2736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51671: BlockingRecv error: recv got EOF from 127.0.98.2:51671 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 03:28:42 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:42 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:42.895531 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:42.890743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.3:57687)
0927 03:28:42.890916 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:42.890927 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:42.891003 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:42.891148 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:42.891149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:42.891335 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:42.891385 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:42.891386 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:42.892135 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:42.892135 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:42.892665 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:42.892666 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:42.893020 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:42.893021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:42.893058 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:42.893604 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:42.893609 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:42.895462 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":106,"threads_started":1}
W20260927 03:28:42.895561 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:42.890830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57687 (local address 127.0.98.62:36343)
0927 03:28:42.890890 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:42.890891 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:42.890992 (+ 101us) server_negotiation.cc:407] Connection header received
0927 03:28:42.891071 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:42.891071 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:42.891085 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:42.891126 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:42.891518 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:42.892060 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:42.892759 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:42.892948 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:42.893012 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:42.895513 (+ 2501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57687: BlockingRecv error: recv got EOF from 127.0.98.3:57687 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 27 03:28:43 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:43 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:43.616061 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:43.611134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.1:43273)
0927 03:28:43.611307 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:43.611321 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:43.611396 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:43.611689 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:43.611691 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:43.611860 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:43.611912 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:43.611914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:43.612548 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:43.612548 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:43.613078 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:43.613079 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:43.613443 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:43.613444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:43.613482 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:43.614123 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:43.614129 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:43.615961 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":77,"threads_started":1}
W20260927 03:28:43.616115 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:43.611280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43273 (local address 127.0.98.62:36343)
0927 03:28:43.611408 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:43.611410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:43.611478 (+ 68us) server_negotiation.cc:407] Connection header received
0927 03:28:43.611512 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:43.611513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:43.611523 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:43.611575 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:43.611986 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:43.612479 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:43.613157 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:43.613384 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:43.613435 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:43.616043 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43273: BlockingRecv error: recv got EOF from 127.0.98.1:43273 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
Sep 27 03:28:43 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:43 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:43.768767 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:43.763810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.2:58841)
0927 03:28:43.764001 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:43.764013 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:43.764120 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:43.764317 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:43.764318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:43.764503 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:43.764546 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:43.764548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:43.765497 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:43.765498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:43.766064 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:43.766066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:43.766442 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:43.766443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:43.766500 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:43.766952 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:43.766956 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:43.768710 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":109,"threads_started":1}
W20260927 03:28:43.768929 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:43.763913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58841 (local address 127.0.98.62:36343)
0927 03:28:43.763989 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:43.763991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:43.764099 (+ 108us) server_negotiation.cc:407] Connection header received
0927 03:28:43.764226 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:43.764227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:43.764246 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:43.764310 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:43.764652 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:43.765417 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:43.766167 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:43.766361 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:43.766442 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:43.768849 (+ 2407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58841: BlockingRecv error: recv got EOF from 127.0.98.2:58841 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 03:28:43 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:43 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:43.903991 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:43.897574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.3:36079)
0927 03:28:43.897797 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:43.897810 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:43.897878 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:43.898062 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:43.898063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:43.898333 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:43.898394 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:43.898397 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:43.899266 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:43.899267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:43.900267 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:43.900271 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:43.900822 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:43.900824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:43.900897 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:43.901540 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:43.901547 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:43.903908 (+ 2361us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":62,"threads_started":1}
W20260927 03:28:43.904240 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:43.897665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36079 (local address 127.0.98.62:36343)
0927 03:28:43.897731 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:43.897733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:43.897881 (+ 148us) server_negotiation.cc:407] Connection header received
0927 03:28:43.897977 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:43.897978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:43.897996 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:43.898051 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:43.898471 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:43.899161 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:43.900365 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:43.900730 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:43.900817 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:43.904027 (+ 3210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36079: BlockingRecv error: recv got EOF from 127.0.98.3:36079 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 03:28:44 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:44 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:44.621083 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:44.616547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.1:52045)
0927 03:28:44.616712 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:44.616723 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:44.616778 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:44.616904 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:44.616905 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:44.617063 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:44.617112 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:44.617114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:44.617858 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:44.617859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:44.618400 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:44.618402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:44.618763 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:44.618763 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:44.618801 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:44.619256 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:44.619260 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:44.621023 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":93,"threads_started":1}
W20260927 03:28:44.621310 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:44.616656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52045 (local address 127.0.98.62:36343)
0927 03:28:44.616782 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:44.616783 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:44.616811 (+ 28us) server_negotiation.cc:407] Connection header received
0927 03:28:44.616831 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:44.616831 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:44.616841 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:44.616891 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:44.617186 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:44.617781 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:44.618476 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:44.618697 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:44.618751 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:44.621227 (+ 2476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52045: BlockingRecv error: recv got EOF from 127.0.98.1:52045 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
Sep 27 03:28:44 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:44 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:44.780010 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:44.769200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.2:46757)
0927 03:28:44.769308 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:44.769327 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:44.769376 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:44.769518 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:44.769519 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:44.769678 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:44.769729 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:44.769731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:44.770412 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:44.770412 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:44.770934 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:44.770936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:44.771279 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:44.771280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:44.771319 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:44.778011 (+ 6692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:44.778016 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:44.779949 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":62,"threads_started":1}
W20260927 03:28:44.780108 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:44.769278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46757 (local address 127.0.98.62:36343)
0927 03:28:44.769349 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:44.769352 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:44.769366 (+ 14us) server_negotiation.cc:407] Connection header received
0927 03:28:44.769440 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:44.769440 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:44.769451 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:44.769503 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:44.769834 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:44.770329 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:44.771019 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:44.771211 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:44.771270 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:44.780044 (+ 8774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46757: BlockingRecv error: recv got EOF from 127.0.98.2:46757 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 03:28:44 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:44 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:44.909907 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:44.904898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.3:34451)
0927 03:28:44.905071 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:44.905087 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:44.905201 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:44.905386 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:44.905387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:44.905578 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:44.905662 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:44.905664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:44.906347 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:44.906348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:44.906885 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:44.906887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:44.907311 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:44.907312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:44.907379 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:44.907968 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:44.907979 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:44.909817 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
W20260927 03:28:44.909988 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:44.905017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34451 (local address 127.0.98.62:36343)
0927 03:28:44.905097 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:44.905099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:44.905176 (+ 77us) server_negotiation.cc:407] Connection header received
0927 03:28:44.905292 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:44.905293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:44.905306 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:44.905379 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:44.905755 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:44.906260 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:44.906992 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:44.907242 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:44.907310 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:44.909923 (+ 2613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34451: BlockingRecv error: recv got EOF from 127.0.98.3:34451 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 03:28:45 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:45 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:45.626398 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:45.621534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.1:36003)
0927 03:28:45.621658 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:45.621670 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:45.621736 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:45.621907 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:45.621907 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:45.622067 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:45.622114 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:45.622116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:45.622763 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:45.622763 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:45.623280 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:45.623281 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:45.623770 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:45.623770 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:45.623813 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:45.624462 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:45.624467 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:45.626338 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
W20260927 03:28:45.626559 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:45.621632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36003 (local address 127.0.98.62:36343)
0927 03:28:45.621768 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:45.621770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:45.621780 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:28:45.621800 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:45.621801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:45.621811 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:45.621879 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:45.622189 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:45.622694 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:45.623382 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:45.623682 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:45.623752 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:45.626448 (+ 2696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36003: BlockingRecv error: recv got EOF from 127.0.98.1:36003 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
Sep 27 03:28:45 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:45 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:45.785470 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:45.780632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.2:49405)
0927 03:28:45.780835 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:45.780847 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:45.780946 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:45.781110 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:45.781111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:45.781292 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:45.781339 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:45.781341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:45.782013 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:45.782014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:45.782544 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:45.782546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:45.782925 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:45.782926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:45.782982 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:45.783512 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:45.783517 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:45.785411 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":120,"threads_started":1}
W20260927 03:28:45.785624 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:45.780742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49405 (local address 127.0.98.62:36343)
0927 03:28:45.780824 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:45.780826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:45.780926 (+ 100us) server_negotiation.cc:407] Connection header received
0927 03:28:45.781008 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:45.781009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:45.781024 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:45.781091 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:45.781424 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:45.781927 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:45.782638 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:45.782841 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:45.782926 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:45.785543 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49405: BlockingRecv error: recv got EOF from 127.0.98.2:49405 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 03:28:45 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:45 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:45.915195 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:45.910480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.3:53123)
0927 03:28:45.910626 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:45.910638 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:45.910700 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:45.910839 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:45.910840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:45.911015 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:45.911059 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:45.911060 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:45.911813 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:45.911814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:45.912334 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:45.912335 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:45.912720 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:45.912721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:45.912783 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:45.913219 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:45.913223 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:45.915131 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
W20260927 03:28:45.915283 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:45.910596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53123 (local address 127.0.98.62:36343)
0927 03:28:45.910642 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:45.910645 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:45.910681 (+ 36us) server_negotiation.cc:407] Connection header received
0927 03:28:45.910759 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:45.910760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:45.910773 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:45.910829 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:45.911144 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:45.911717 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:45.912429 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:45.912616 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:45.912721 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:45.915212 (+ 2491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53123: BlockingRecv error: recv got EOF from 127.0.98.3:53123 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 27 03:28:46 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:46 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:46.631636 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:46.626999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.1:43131)
0927 03:28:46.627160 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:46.627171 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:46.627226 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:46.627378 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:46.627379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:46.627594 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:46.627643 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:46.627645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:46.628267 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:46.628268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:46.628771 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:46.628773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:46.629076 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:46.629076 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:46.629113 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:46.629540 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:46.629544 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:46.631549 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":84,"threads_started":1}
W20260927 03:28:46.631688 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:46.627086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43131 (local address 127.0.98.62:36343)
0927 03:28:46.627226 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:46.627228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:46.627259 (+ 31us) server_negotiation.cc:407] Connection header received
0927 03:28:46.627288 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:46.627288 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:46.627297 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:46.627359 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:46.627717 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:46.628205 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:46.628837 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:46.629016 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:46.629063 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:46.631635 (+ 2572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43131: BlockingRecv error: recv got EOF from 127.0.98.1:43131 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
Sep 27 03:28:46 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:46 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:46.790589 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:46.786041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.2:48229)
0927 03:28:46.786223 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:46.786235 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:46.786306 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:46.786471 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:46.786473 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:46.786644 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:46.786694 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:46.786696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:46.787335 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:46.787336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:46.787895 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:46.787897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:46.788273 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:46.788274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:46.788319 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:46.788851 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:46.788855 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:46.790531 (+ 1676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":107,"threads_started":1}
W20260927 03:28:46.790684 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:46.786098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48229 (local address 127.0.98.62:36343)
0927 03:28:46.786171 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:46.786173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:46.786308 (+ 135us) server_negotiation.cc:407] Connection header received
0927 03:28:46.786368 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:46.786369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:46.786383 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:46.786456 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:46.786764 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:46.787260 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:46.787986 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:46.788195 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:46.788252 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:46.790619 (+ 2367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48229: BlockingRecv error: recv got EOF from 127.0.98.2:48229 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 03:28:46 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:46 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:46.920518 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:46.915736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.3:59137)
0927 03:28:46.915895 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:46.915925 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:46.915989 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:46.916204 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:46.916205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:46.916402 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:46.916449 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:46.916451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:46.917132 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:46.917133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:46.917652 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:46.917653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:46.918046 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:46.918048 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:46.918111 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:46.918584 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:46.918588 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:46.920424 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":108,"threads_started":1}
W20260927 03:28:46.920567 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:46.915806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59137 (local address 127.0.98.62:36343)
0927 03:28:46.915890 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:46.915892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:46.915998 (+ 106us) server_negotiation.cc:407] Connection header received
0927 03:28:46.916080 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:46.916080 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:46.916093 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:46.916188 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:46.916540 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:46.917045 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:46.917747 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:46.917954 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:46.918040 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:46.920503 (+ 2463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59137: BlockingRecv error: recv got EOF from 127.0.98.3:59137 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 03:28:47 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:47 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:47.636747 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:47.632122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.1:60561)
0927 03:28:47.632279 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:47.632298 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:47.632344 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:47.632525 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:47.632526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:47.632698 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:47.632747 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:47.632748 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:47.633370 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:47.633371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:47.633898 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:47.633899 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:47.634237 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:47.634238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:47.634275 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:47.634951 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:47.634956 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:47.636687 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":99,"threads_started":1}
W20260927 03:28:47.636857 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:47.632170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60561 (local address 127.0.98.62:36343)
0927 03:28:47.632398 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:47.632400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:47.632408 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:28:47.632438 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:47.632438 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:47.632448 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:47.632510 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:47.632811 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:47.633300 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:47.633967 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:47.634182 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:47.634230 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:47.636783 (+ 2553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60561: BlockingRecv error: recv got EOF from 127.0.98.1:60561 (error 108)
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":179,"threads_started":1}
Sep 27 03:28:47 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:47 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:47.796011 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:47.791102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.2:36453)
0927 03:28:47.791268 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:47.791282 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:47.791365 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:47.791575 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:47.791576 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:47.791793 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:47.791840 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:47.791843 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:47.792527 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:47.792528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:47.793059 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:47.793061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:47.793448 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:47.793449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:47.793506 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:47.794077 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:47.794083 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:47.795946 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":86,"threads_started":1}
W20260927 03:28:47.796077 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:47.791224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36453 (local address 127.0.98.62:36343)
0927 03:28:47.791292 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:47.791294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:47.791339 (+ 45us) server_negotiation.cc:407] Connection header received
0927 03:28:47.791482 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:47.791485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:47.791503 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:47.791568 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:47.791928 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:47.792435 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:47.793167 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:47.793374 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:47.793451 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:47.796032 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36453: BlockingRecv error: recv got EOF from 127.0.98.2:36453 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 03:28:47 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:47 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:47.926011 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:47.921080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.3:41161)
0927 03:28:47.921280 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:47.921292 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:47.921385 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:47.921535 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:47.921536 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:47.921736 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:47.921779 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:47.921781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:47.922456 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:47.922457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:47.922982 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:47.922983 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:47.923354 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:47.923355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:47.923463 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:47.924042 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:47.924048 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:47.925901 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":107,"threads_started":1}
W20260927 03:28:47.926041 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:47.921151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41161 (local address 127.0.98.62:36343)
0927 03:28:47.921251 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:47.921253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:47.921367 (+ 114us) server_negotiation.cc:407] Connection header received
0927 03:28:47.921443 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:47.921444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:47.921459 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:47.921529 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:47.921864 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:47.922365 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:47.923069 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:47.923268 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:47.923353 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:47.925977 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41161: BlockingRecv error: recv got EOF from 127.0.98.3:41161 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 03:28:48 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:48 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:48.642347 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:48.637332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.1:43203)
0927 03:28:48.637474 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:48.637492 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:48.637581 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:48.637740 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:48.637741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:48.637909 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:48.637960 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:48.637962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:48.638702 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:48.638703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:48.639295 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:48.639297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:48.639653 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:48.639653 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:48.639698 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:48.640351 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:48.640357 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:48.642274 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
W20260927 03:28:48.642447 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:48.637384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43203 (local address 127.0.98.62:36343)
0927 03:28:48.637542 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:48.637549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:48.637560 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:28:48.637644 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:48.637645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:48.637655 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:48.637721 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:48.638072 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:48.638598 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:48.639378 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:48.639587 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:48.639646 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:48.642373 (+ 2727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43203: BlockingRecv error: recv got EOF from 127.0.98.1:43203 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":118,"threads_started":1}
Sep 27 03:28:48 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:48 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:48.801272 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:48.796525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.2:35417)
0927 03:28:48.796653 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:48.796676 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:48.796727 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:48.796930 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:48.796932 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:48.797113 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:48.797163 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:48.797164 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:48.797862 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:48.797864 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:48.798392 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:48.798393 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:48.798761 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:48.798762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:48.798820 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:48.799332 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:48.799337 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:48.801206 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
W20260927 03:28:48.801411 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:48.796727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35417 (local address 127.0.98.62:36343)
0927 03:28:48.796807 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:48.796809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:48.796823 (+ 14us) server_negotiation.cc:407] Connection header received
0927 03:28:48.796846 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:48.796847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:48.796857 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:48.796924 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:48.797250 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:48.797758 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:48.798486 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:48.798684 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:48.798763 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:48.801327 (+ 2564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35417: BlockingRecv error: recv got EOF from 127.0.98.2:35417 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 03:28:48 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:48 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:48.932075 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:48.926607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.3:45275)
0927 03:28:48.926781 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:48.926796 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:48.926879 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:48.927025 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:48.927025 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:48.927294 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:48.927378 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:48.927381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:48.928321 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:48.928322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:48.928879 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:48.928882 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:48.929290 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:48.929292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:48.929344 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:48.929983 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:48.929988 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:48.931974 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":71,"threads_started":1}
W20260927 03:28:48.932125 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:48.926728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45275 (local address 127.0.98.62:36343)
0927 03:28:48.926800 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:48.926802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:48.926874 (+ 72us) server_negotiation.cc:407] Connection header received
0927 03:28:48.926935 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:48.926936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:48.926949 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:48.927000 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:48.927537 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:48.928213 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:48.928995 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:48.929195 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:48.929254 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:48.932056 (+ 2802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45275: BlockingRecv error: recv got EOF from 127.0.98.3:45275 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 03:28:49 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:49 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:49.647511 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:49.642882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.1:39451)
0927 03:28:49.643030 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:49.643041 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:49.643095 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:49.643265 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:49.643266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:49.643459 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:49.643509 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:49.643511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:49.644212 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:49.644213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:49.644746 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:49.644748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:49.645090 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:49.645091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:49.645139 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:49.645642 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:49.645647 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:49.647449 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
W20260927 03:28:49.647753 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:49.642995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39451 (local address 127.0.98.62:36343)
0927 03:28:49.643129 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:49.643131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:49.643141 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:28:49.643162 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:49.643163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:49.643173 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:49.643243 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:49.643599 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:49.644133 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:49.644830 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:49.645023 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:49.645078 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:49.647608 (+ 2530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39451: BlockingRecv error: recv got EOF from 127.0.98.1:39451 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
Sep 27 03:28:49 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:49 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:49.811879 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:49.801779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.2:35133)
0927 03:28:49.801948 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:49.801968 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:49.802045 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:49.802179 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:49.802180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:49.802348 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:49.802396 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:49.802398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:49.803045 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:49.803046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:49.803595 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:49.803597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:49.803932 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:49.803933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:49.803979 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:49.809640 (+ 5661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:49.809645 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:49.811818 (+ 2173us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":58,"threads_started":1}
W20260927 03:28:49.811990 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:49.801903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35133 (local address 127.0.98.62:36343)
0927 03:28:49.801972 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:49.801974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:49.802039 (+ 65us) server_negotiation.cc:407] Connection header received
0927 03:28:49.802093 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:49.802094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:49.802106 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:49.802155 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:49.802459 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:49.802953 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:49.803666 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:49.803853 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:49.803905 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:49.811921 (+ 8016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35133: BlockingRecv error: recv got EOF from 127.0.98.2:35133 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 03:28:49 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:49 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:49.937599 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:49.932619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.3:32787)
0927 03:28:49.932793 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:49.932807 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:49.932882 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:49.933019 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:49.933020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:49.933185 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:49.933232 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:49.933233 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:49.933872 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:49.933873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:49.934417 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:49.934419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:49.934845 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:49.934846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:49.934889 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:49.935452 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:49.935457 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:49.937516 (+ 2059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":70,"threads_started":1}
W20260927 03:28:49.937695 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:49.932746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32787 (local address 127.0.98.62:36343)
0927 03:28:49.932812 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:49.932814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:49.932876 (+ 62us) server_negotiation.cc:407] Connection header received
0927 03:28:49.932937 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:49.932938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:49.932950 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:49.933001 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:49.933290 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:49.933791 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:49.934557 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:49.934775 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:49.934836 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:49.937603 (+ 2767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:32787: BlockingRecv error: recv got EOF from 127.0.98.3:32787 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 03:28:50 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:50 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:50.652843 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:50.648103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.1:40261)
0927 03:28:50.648242 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:50.648254 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:50.648311 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:50.648528 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:50.648528 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:50.648687 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:50.648753 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:50.648755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:50.649416 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:50.649417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:50.649940 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:50.649941 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:50.650294 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:50.650295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:50.650349 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:50.650944 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:50.650949 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:50.652781 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
W20260927 03:28:50.652937 9624 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:50.648221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40261 (local address 127.0.98.62:36343)
0927 03:28:50.648348 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:50.648350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:50.648360 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:28:50.648405 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:50.648406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:50.648420 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:50.648529 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:50.648841 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:50.649349 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:50.650016 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:50.650207 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:50.650260 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:50.652865 (+ 2605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40261: BlockingRecv error: recv got EOF from 127.0.98.1:40261 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
Sep 27 03:28:50 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:28:50 dist-test-slave-j0hf krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:28:50.817554 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:50.812440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36343 (local address 127.0.98.2:51573)
0927 03:28:50.812567 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:50.812579 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:50.812648 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:50.812805 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:50.812805 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:50.812986 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:50.813035 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:50.813037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:50.813760 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:50.813760 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:50.814272 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:50.814273 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:50.814628 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:50.814629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:50.814686 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:50.815307 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:50.815312 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:50.817485 (+ 2173us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":55,"threads_started":1}
W20260927 03:28:50.817708 9624 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:50.812542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51573 (local address 127.0.98.62:36343)
0927 03:28:50.812661 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:50.812663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:50.812672 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:28:50.812698 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:50.812699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:50.812709 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:50.812782 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:50.813131 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:50.813661 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:50.814347 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:50.814534 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:50.814589 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:50.817611 (+ 3022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51573: BlockingRecv error: recv got EOF from 127.0.98.2:51573 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 03:28:50.842968 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 9149
I20260927 03:28:50.847703 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 9301
I20260927 03:28:50.851728 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 9434
I20260927 03:28:50.855922 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 9079
2026-09-27T03:28:50Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17467 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T03:28:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:28:50Z Disabled control of system clock
I20260927 03:28:50.883178 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38303
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38303
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 03:28:50.962260 9629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:50.962458 9629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:50.962478 9629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:50.963727 9629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:28:50.963778 9629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:50.963791 9629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:50.963804 9629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:28:50.963816 9629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:28:50.965164 9629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38303
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:50.965417 9629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:50.965706 9629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:50.970949 9637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:50.970938 9635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:50.970966 9629 server_base.cc:1034] running on GCE node
W20260927 03:28:50.970939 9634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:50.971477 9629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:50.971822 9629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:50.972972 9629 hybrid_clock.cc:648] HybridClock initialized: now 1790479730972954 us; error 38 us; skew 500 ppm
I20260927 03:28:50.974253 9629 webserver.cc:460] Webserver started at http://127.0.98.62:35187/ using document root <none> and password file <none>
I20260927 03:28:50.974488 9629 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:50.974547 9629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:50.974720 9629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:50.976104 9629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "9553d6039aea4b17bc76ea6ceefb7dee"
format_stamp: "Formatted at 2026-09-27 03:28:50 on dist-test-slave-j0hf"
server_key: "c8c53c72ec8f9e3dda8f2540ddb31c28"
server_key_iv: "b55606cee1abd3319f618a4919efa4ea"
server_key_version: "encryptionkey@0"
I20260927 03:28:50.976513 9629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "9553d6039aea4b17bc76ea6ceefb7dee"
format_stamp: "Formatted at 2026-09-27 03:28:50 on dist-test-slave-j0hf"
server_key: "c8c53c72ec8f9e3dda8f2540ddb31c28"
server_key_iv: "b55606cee1abd3319f618a4919efa4ea"
server_key_version: "encryptionkey@0"
I20260927 03:28:50.977854 9629 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:28:50.978515 9643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:50.978674 9629 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:50.978775 9629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "9553d6039aea4b17bc76ea6ceefb7dee"
format_stamp: "Formatted at 2026-09-27 03:28:50 on dist-test-slave-j0hf"
server_key: "c8c53c72ec8f9e3dda8f2540ddb31c28"
server_key_iv: "b55606cee1abd3319f618a4919efa4ea"
server_key_version: "encryptionkey@0"
I20260927 03:28:50.978860 9629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:50.998456 9629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:50.998805 9629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:50.998976 9629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:51.003002 9629 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38303
I20260927 03:28:51.003046 9695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38303 every 8 connection(s)
I20260927 03:28:51.003404 9629 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:28:51.004143 9696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:51.006774 9696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9553d6039aea4b17bc76ea6ceefb7dee: Bootstrap starting.
I20260927 03:28:51.007454 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 9629
I20260927 03:28:51.007556 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:28:51.007560 9696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9553d6039aea4b17bc76ea6ceefb7dee: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:51.007786 392 external_mini_cluster.cc:1428] Setting key e2ef1658c6a5b417f0a50f6af7993602
I20260927 03:28:51.007894 9696 log.cc:824] T 00000000000000000000000000000000 P 9553d6039aea4b17bc76ea6ceefb7dee: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:51.008858 9696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9553d6039aea4b17bc76ea6ceefb7dee: No bootstrap required, opened a new log
I20260927 03:28:51.011080 9696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9553d6039aea4b17bc76ea6ceefb7dee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9553d6039aea4b17bc76ea6ceefb7dee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38303 } }
I20260927 03:28:51.011253 9696 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9553d6039aea4b17bc76ea6ceefb7dee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:51.011288 9696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9553d6039aea4b17bc76ea6ceefb7dee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553d6039aea4b17bc76ea6ceefb7dee, State: Initialized, Role: FOLLOWER
I20260927 03:28:51.011406 9696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9553d6039aea4b17bc76ea6ceefb7dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9553d6039aea4b17bc76ea6ceefb7dee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38303 } }
I20260927 03:28:51.011514 9696 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9553d6039aea4b17bc76ea6ceefb7dee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:28:51.011547 9696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9553d6039aea4b17bc76ea6ceefb7dee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:28:51.011593 9696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9553d6039aea4b17bc76ea6ceefb7dee [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:51.012198 9696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9553d6039aea4b17bc76ea6ceefb7dee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9553d6039aea4b17bc76ea6ceefb7dee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38303 } }
I20260927 03:28:51.012310 9696 leader_election.cc:302] T 00000000000000000000000000000000 P 9553d6039aea4b17bc76ea6ceefb7dee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9553d6039aea4b17bc76ea6ceefb7dee; no voters:
I20260927 03:28:51.012535 9696 leader_election.cc:288] T 00000000000000000000000000000000 P 9553d6039aea4b17bc76ea6ceefb7dee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:28:51.012583 9701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9553d6039aea4b17bc76ea6ceefb7dee [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:51.012785 9701 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9553d6039aea4b17bc76ea6ceefb7dee [term 1 LEADER]: Becoming Leader. State: Replica: 9553d6039aea4b17bc76ea6ceefb7dee, State: Running, Role: LEADER
I20260927 03:28:51.012833 9696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9553d6039aea4b17bc76ea6ceefb7dee [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:28:51.012944 9701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9553d6039aea4b17bc76ea6ceefb7dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9553d6039aea4b17bc76ea6ceefb7dee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38303 } }
I20260927 03:28:51.013288 9703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9553d6039aea4b17bc76ea6ceefb7dee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9553d6039aea4b17bc76ea6ceefb7dee. Latest consensus state: current_term: 1 leader_uuid: "9553d6039aea4b17bc76ea6ceefb7dee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9553d6039aea4b17bc76ea6ceefb7dee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38303 } } }
I20260927 03:28:51.013422 9702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9553d6039aea4b17bc76ea6ceefb7dee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9553d6039aea4b17bc76ea6ceefb7dee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9553d6039aea4b17bc76ea6ceefb7dee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38303 } } }
I20260927 03:28:51.013552 9702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9553d6039aea4b17bc76ea6ceefb7dee [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:51.013466 9703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9553d6039aea4b17bc76ea6ceefb7dee [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:51.013883 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:51.008397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38414 (local address 127.0.98.62:38303)
0927 03:28:51.008622 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:51.008631 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:51.008673 (+ 42us) server_negotiation.cc:407] Connection header received
0927 03:28:51.009194 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:51.009202 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:51.009266 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:51.009392 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:51.009633 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:51.010510 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:51.011713 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:51.012120 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:51.012213 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:51.013008 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:51.013012 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:51.013012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:28:51.013254 (+ 242us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:51.013294 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:28:51.013309 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:51.013386 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:51.013591 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 03:28:51.013677 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
I20260927 03:28:51.013885 9710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:28:51.014530 9710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:28:51.016099 9710 catalog_manager.cc:1347] Generated new cluster ID: f10eae538c4a4bc6bb3fa4b585c30eca
I20260927 03:28:51.016155 9710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:28:51.033176 9710 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:28:51.033720 9710 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:28:51.037442 9710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9553d6039aea4b17bc76ea6ceefb7dee: Generated new TSK 0
I20260927 03:28:51.037640 9710 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:28:51.039742 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38303
--builtin_ntp_servers=127.0.98.20:37271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 03:28:51.121145 9720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:51.121385 9720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:51.121411 9720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:51.122684 9720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:51.122743 9720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:51.122773 9720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:28:51.124230 9720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:51.124498 9720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:51.124836 9720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:51.130921 9728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:51.130894 9725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:51.130903 9726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:51.131152 9720 server_base.cc:1034] running on GCE node
I20260927 03:28:51.131381 9720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:51.131676 9720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:51.132829 9720 hybrid_clock.cc:648] HybridClock initialized: now 1790479731132813 us; error 31 us; skew 500 ppm
I20260927 03:28:51.134136 9720 webserver.cc:460] Webserver started at http://127.0.98.1:42857/ using document root <none> and password file <none>
I20260927 03:28:51.134398 9720 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:51.134471 9720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:51.134613 9720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:51.135814 9720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/data/instance:
uuid: "6d325151e8cd4b52a38de5b9dda016f4"
format_stamp: "Formatted at 2026-09-27 03:28:51 on dist-test-slave-j0hf"
server_key: "4a4ce6356ee8ae4a5b7678aee696cca9"
server_key_iv: "4353e2fb0401448f4334e5e5a972f03b"
server_key_version: "encryptionkey@0"
I20260927 03:28:51.136186 9720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6d325151e8cd4b52a38de5b9dda016f4"
format_stamp: "Formatted at 2026-09-27 03:28:51 on dist-test-slave-j0hf"
server_key: "4a4ce6356ee8ae4a5b7678aee696cca9"
server_key_iv: "4353e2fb0401448f4334e5e5a972f03b"
server_key_version: "encryptionkey@0"
I20260927 03:28:51.137566 9720 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:51.138396 9734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:51.138648 9720 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:51.138756 9720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "6d325151e8cd4b52a38de5b9dda016f4"
format_stamp: "Formatted at 2026-09-27 03:28:51 on dist-test-slave-j0hf"
server_key: "4a4ce6356ee8ae4a5b7678aee696cca9"
server_key_iv: "4353e2fb0401448f4334e5e5a972f03b"
server_key_version: "encryptionkey@0"
I20260927 03:28:51.138832 9720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:51.157128 9720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:51.157440 9720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:51.157601 9720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:51.157876 9720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:51.158245 9720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:51.158280 9720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:51.158303 9720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:51.158352 9720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:51.164110 9720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39445
I20260927 03:28:51.164220 9847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39445 every 8 connection(s)
I20260927 03:28:51.164539 9720 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:28:51.164795 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 9720
I20260927 03:28:51.164886 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:28:51.165108 392 external_mini_cluster.cc:1428] Setting key 6066cc1f44c28460715c5284ccbce683
I20260927 03:28:51.166124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38303
--builtin_ntp_servers=127.0.98.20:37271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 03:28:51.170670 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:51.165315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49847 (local address 127.0.98.62:38303)
0927 03:28:51.165384 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:51.165386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:51.165899 (+ 513us) server_negotiation.cc:407] Connection header received
0927 03:28:51.166505 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:51.166506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:51.166523 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:51.166596 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:51.167478 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:51.167981 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:51.169064 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:51.169384 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:51.169459 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:51.170338 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:51.170340 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:51.170340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:28:51.170347 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:51.170364 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:28:51.170369 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:51.170435 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:51.170521 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 03:28:51.170549 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:28:51.170879 9850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:51.165292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38303 (local address 127.0.98.1:49847)
0927 03:28:51.165703 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:51.165793 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:51.166406 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:51.166650 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:51.166653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:51.166661 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:51.167322 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:51.167331 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:51.168081 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:51.168082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:51.168974 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:51.168976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:51.169480 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:51.169481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:51.169597 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:51.170192 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:28:51.170193 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:51.170222 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:28:51.170249 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:51.170441 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:51.170444 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:51.170447 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:28:51.170500 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 03:28:51.170583 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260927 03:28:51.171377 9848 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38303
I20260927 03:28:51.171497 9848 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:51.171682 9848 heartbeater.cc:511] Master 127.0.98.62:38303 requested a full tablet report, sending...
I20260927 03:28:51.172204 9660 ts_manager.cc:194] Registered new tserver with Master: 6d325151e8cd4b52a38de5b9dda016f4 (127.0.98.1:39445)
I20260927 03:28:51.172834 9660 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:49847
W20260927 03:28:51.246977 9851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:51.247210 9851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:51.247254 9851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:51.248457 9851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:51.248522 9851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:51.248562 9851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:28:51.249855 9851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:51.250111 9851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:51.250407 9851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:51.255430 9859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:51.255381 9856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:51.255479 9851 server_base.cc:1034] running on GCE node
W20260927 03:28:51.255398 9857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:51.255853 9851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:51.256053 9851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:51.257189 9851 hybrid_clock.cc:648] HybridClock initialized: now 1790479731257172 us; error 32 us; skew 500 ppm
I20260927 03:28:51.258462 9851 webserver.cc:460] Webserver started at http://127.0.98.2:36099/ using document root <none> and password file <none>
I20260927 03:28:51.258690 9851 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:51.258752 9851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:51.258927 9851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:51.260030 9851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/data/instance:
uuid: "54c275b9d7544fee9f26177ebffdc0e0"
format_stamp: "Formatted at 2026-09-27 03:28:51 on dist-test-slave-j0hf"
server_key: "ed0b72d4fb9459f80fd7a320bf6e3624"
server_key_iv: "c53c37deeb08565e77070e1bb86e14ec"
server_key_version: "encryptionkey@0"
I20260927 03:28:51.260403 9851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/wal/instance:
uuid: "54c275b9d7544fee9f26177ebffdc0e0"
format_stamp: "Formatted at 2026-09-27 03:28:51 on dist-test-slave-j0hf"
server_key: "ed0b72d4fb9459f80fd7a320bf6e3624"
server_key_iv: "c53c37deeb08565e77070e1bb86e14ec"
server_key_version: "encryptionkey@0"
I20260927 03:28:51.261749 9851 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 03:28:51.262410 9865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:51.262581 9851 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:51.262662 9851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "54c275b9d7544fee9f26177ebffdc0e0"
format_stamp: "Formatted at 2026-09-27 03:28:51 on dist-test-slave-j0hf"
server_key: "ed0b72d4fb9459f80fd7a320bf6e3624"
server_key_iv: "c53c37deeb08565e77070e1bb86e14ec"
server_key_version: "encryptionkey@0"
I20260927 03:28:51.262715 9851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:51.278177 9851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:51.278478 9851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:51.278581 9851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:51.278775 9851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:51.279130 9851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:51.279165 9851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:51.279198 9851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:51.279211 9851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:51.285643 9851 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40661
I20260927 03:28:51.285759 9978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40661 every 8 connection(s)
I20260927 03:28:51.286069 9851 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:28:51.291285 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:51.286871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46945 (local address 127.0.98.62:38303)
0927 03:28:51.286927 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:51.286929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:51.287172 (+ 243us) server_negotiation.cc:407] Connection header received
0927 03:28:51.287766 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:51.287767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:51.287777 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:51.287831 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:51.288482 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:51.288972 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:51.289972 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:51.290161 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:51.290218 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:51.290987 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:51.290989 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:51.290989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:28:51.290993 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:51.291007 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:28:51.291011 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:51.291053 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:51.291204 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 03:28:51.291225 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 03:28:51.291278 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 9851
I20260927 03:28:51.291368 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:28:51.291591 9981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:51.286865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38303 (local address 127.0.98.2:46945)
0927 03:28:51.287046 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:51.287080 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:51.287670 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:51.287887 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:51.287891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:51.287899 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:51.288405 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:51.288409 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:51.289072 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:51.289073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:51.289882 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:51.289884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:51.290240 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:51.290241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:51.290352 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:51.290853 (+ 501us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:28:51.290854 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:51.290881 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:28:51.290905 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:51.291112 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:51.291117 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:51.291120 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:28:51.291208 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 03:28:51.291325 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":96,"threads_started":1}
I20260927 03:28:51.291599 392 external_mini_cluster.cc:1428] Setting key c72158fed1be73d225fd890a95441c0e
I20260927 03:28:51.292179 9979 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38303
I20260927 03:28:51.292297 9979 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:51.292495 9979 heartbeater.cc:511] Master 127.0.98.62:38303 requested a full tablet report, sending...
I20260927 03:28:51.292794 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38303
--builtin_ntp_servers=127.0.98.20:37271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 03:28:51.292918 9659 ts_manager.cc:194] Registered new tserver with Master: 54c275b9d7544fee9f26177ebffdc0e0 (127.0.98.2:40661)
I20260927 03:28:51.293445 9659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:46945
W20260927 03:28:51.374454 9982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:51.374698 9982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:51.374734 9982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:51.375941 9982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:51.376001 9982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:51.376042 9982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:28:51.377401 9982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:51.377676 9982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:51.377970 9982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:51.383131 9987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:51.383160 9990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:51.383186 9988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:51.383307 9982 server_base.cc:1034] running on GCE node
I20260927 03:28:51.383795 9982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:51.384013 9982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:51.385174 9982 hybrid_clock.cc:648] HybridClock initialized: now 1790479731385160 us; error 26 us; skew 500 ppm
I20260927 03:28:51.386519 9982 webserver.cc:460] Webserver started at http://127.0.98.3:33931/ using document root <none> and password file <none>
I20260927 03:28:51.386819 9982 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:51.386875 9982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:51.387056 9982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:51.388274 9982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/data/instance:
uuid: "c4d52b0a881d4186afff776d070554ee"
format_stamp: "Formatted at 2026-09-27 03:28:51 on dist-test-slave-j0hf"
server_key: "36998353de73df90674df78a02a7b60b"
server_key_iv: "715ef0ce351cb2698541f526d484da25"
server_key_version: "encryptionkey@0"
I20260927 03:28:51.388657 9982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c4d52b0a881d4186afff776d070554ee"
format_stamp: "Formatted at 2026-09-27 03:28:51 on dist-test-slave-j0hf"
server_key: "36998353de73df90674df78a02a7b60b"
server_key_iv: "715ef0ce351cb2698541f526d484da25"
server_key_version: "encryptionkey@0"
I20260927 03:28:51.390103 9982 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:28:51.390925 9996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:51.391129 9982 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:28:51.391204 9982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "c4d52b0a881d4186afff776d070554ee"
format_stamp: "Formatted at 2026-09-27 03:28:51 on dist-test-slave-j0hf"
server_key: "36998353de73df90674df78a02a7b60b"
server_key_iv: "715ef0ce351cb2698541f526d484da25"
server_key_version: "encryptionkey@0"
I20260927 03:28:51.391299 9982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:51.418898 9982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:51.419247 9982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:51.419404 9982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:51.419724 9982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:51.420090 9982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:51.420146 9982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:51.420181 9982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:51.420208 9982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:51.425670 9982 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44419
I20260927 03:28:51.425755 10109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44419 every 8 connection(s)
I20260927 03:28:51.426095 9982 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:28:51.428114 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 9982
I20260927 03:28:51.428251 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:28:51.428444 392 external_mini_cluster.cc:1428] Setting key 1cb3a979f459f5ba4d67dda0288d9c21
I20260927 03:28:51.431715 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:51.426923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50181 (local address 127.0.98.62:38303)
0927 03:28:51.426989 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:51.426991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:51.427223 (+ 232us) server_negotiation.cc:407] Connection header received
0927 03:28:51.427883 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:51.427885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:51.427899 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:51.427953 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:51.428791 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:51.429290 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:51.430333 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:51.430531 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:51.430610 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:51.431447 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:51.431448 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:51.431449 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:28:51.431454 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:51.431469 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:28:51.431473 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:51.431521 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:51.431601 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 03:28:51.431628 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 03:28:51.431931 10112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:51.426896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38303 (local address 127.0.98.3:50181)
0927 03:28:51.427075 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:51.427110 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:51.427760 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:51.428028 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:51.428032 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:51.428042 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:51.428690 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:51.428695 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:51.429391 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:51.429393 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:51.430227 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:51.430229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:51.430617 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:51.430617 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:51.430786 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:51.431288 (+ 502us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:28:51.431289 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:51.431317 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:28:51.431339 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:51.431521 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:51.431527 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:51.431530 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:28:51.431604 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 03:28:51.431704 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":65,"threads_started":1}
I20260927 03:28:51.432381 10110 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38303
I20260927 03:28:51.432472 10110 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:51.432611 10110 heartbeater.cc:511] Master 127.0.98.62:38303 requested a full tablet report, sending...
I20260927 03:28:51.432928 9659 ts_manager.cc:194] Registered new tserver with Master: c4d52b0a881d4186afff776d070554ee (127.0.98.3:44419)
I20260927 03:28:51.433301 9659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:50181
I20260927 03:28:51.440095 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 03:28:51.441931 9700 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:51.441384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38420 (local address 127.0.98.62:38303)
0927 03:28:51.441461 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:51.441463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:51.441579 (+ 116us) server_negotiation.cc:407] Connection header received
0927 03:28:51.441677 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:51.441678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:51.441688 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:51.441733 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:51.441866 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38420: BlockingRecv error: recv got EOF from 127.0.0.1:38420 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 03:28:51.442003 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 9720
I20260927 03:28:51.446597 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 9851
I20260927 03:28:51.450528 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 9982
I20260927 03:28:51.454804 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 9629
2026-09-27T03:28:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (598 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T03:28:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:28:51Z Disabled control of system clock
I20260927 03:28:51.481647 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35079
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 03:28:51.563155 10121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:51.563370 10121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:51.563391 10121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:51.564587 10121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:28:51.564631 10121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:51.564644 10121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:51.564657 10121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:28:51.564669 10121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:28:51.566124 10121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35079
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:51.566385 10121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:51.566671 10121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:51.572109 10126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:51.572093 10129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:51.572116 10127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:51.572408 10121 server_base.cc:1034] running on GCE node
I20260927 03:28:51.572636 10121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:51.572902 10121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:51.574036 10121 hybrid_clock.cc:648] HybridClock initialized: now 1790479731574018 us; error 34 us; skew 500 ppm
I20260927 03:28:51.575403 10121 webserver.cc:460] Webserver started at http://127.0.98.62:39379/ using document root <none> and password file <none>
I20260927 03:28:51.575703 10121 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:51.575750 10121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:51.575915 10121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:51.577009 10121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "73ff95eb815c438e9f94ddf171b78ea4"
format_stamp: "Formatted at 2026-09-27 03:28:51 on dist-test-slave-j0hf"
server_key: "45f250351994ad49fad01b9a339e12bd"
server_key_iv: "ef55212444e606651b40ad66dea21fb9"
server_key_version: "encryptionkey@0"
I20260927 03:28:51.577371 10121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "73ff95eb815c438e9f94ddf171b78ea4"
format_stamp: "Formatted at 2026-09-27 03:28:51 on dist-test-slave-j0hf"
server_key: "45f250351994ad49fad01b9a339e12bd"
server_key_iv: "ef55212444e606651b40ad66dea21fb9"
server_key_version: "encryptionkey@0"
I20260927 03:28:51.578662 10121 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:28:51.579357 10135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:51.579531 10121 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:28:51.579635 10121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "73ff95eb815c438e9f94ddf171b78ea4"
format_stamp: "Formatted at 2026-09-27 03:28:51 on dist-test-slave-j0hf"
server_key: "45f250351994ad49fad01b9a339e12bd"
server_key_iv: "ef55212444e606651b40ad66dea21fb9"
server_key_version: "encryptionkey@0"
I20260927 03:28:51.579735 10121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:51.583009 10121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:51.583256 10121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:51.583407 10121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:51.587093 10121 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35079
I20260927 03:28:51.587122 10187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35079 every 8 connection(s)
I20260927 03:28:51.587531 10121 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:28:51.588191 10188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:51.590830 10188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73ff95eb815c438e9f94ddf171b78ea4: Bootstrap starting.
I20260927 03:28:51.591403 10188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73ff95eb815c438e9f94ddf171b78ea4: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:51.591758 10188 log.cc:824] T 00000000000000000000000000000000 P 73ff95eb815c438e9f94ddf171b78ea4: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:51.592401 10188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73ff95eb815c438e9f94ddf171b78ea4: No bootstrap required, opened a new log
I20260927 03:28:51.593928 10188 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73ff95eb815c438e9f94ddf171b78ea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ff95eb815c438e9f94ddf171b78ea4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35079 } }
I20260927 03:28:51.594058 10188 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73ff95eb815c438e9f94ddf171b78ea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:51.594106 10188 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73ff95eb815c438e9f94ddf171b78ea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73ff95eb815c438e9f94ddf171b78ea4, State: Initialized, Role: FOLLOWER
I20260927 03:28:51.594259 10188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73ff95eb815c438e9f94ddf171b78ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ff95eb815c438e9f94ddf171b78ea4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35079 } }
I20260927 03:28:51.594343 10188 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73ff95eb815c438e9f94ddf171b78ea4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:28:51.594394 10188 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73ff95eb815c438e9f94ddf171b78ea4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:28:51.594442 10188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73ff95eb815c438e9f94ddf171b78ea4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:51.594964 10188 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73ff95eb815c438e9f94ddf171b78ea4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ff95eb815c438e9f94ddf171b78ea4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35079 } }
I20260927 03:28:51.595070 10188 leader_election.cc:302] T 00000000000000000000000000000000 P 73ff95eb815c438e9f94ddf171b78ea4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73ff95eb815c438e9f94ddf171b78ea4; no voters:
I20260927 03:28:51.595243 10188 leader_election.cc:288] T 00000000000000000000000000000000 P 73ff95eb815c438e9f94ddf171b78ea4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:28:51.595292 10191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73ff95eb815c438e9f94ddf171b78ea4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:51.595475 10191 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73ff95eb815c438e9f94ddf171b78ea4 [term 1 LEADER]: Becoming Leader. State: Replica: 73ff95eb815c438e9f94ddf171b78ea4, State: Running, Role: LEADER
I20260927 03:28:51.595530 10188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73ff95eb815c438e9f94ddf171b78ea4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:28:51.595561 10191 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73ff95eb815c438e9f94ddf171b78ea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ff95eb815c438e9f94ddf171b78ea4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35079 } }
I20260927 03:28:51.595808 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 10121
I20260927 03:28:51.595894 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:28:51.596089 392 external_mini_cluster.cc:1428] Setting key 6fd87a1f33be8763d0fa31b019b43897
I20260927 03:28:51.596820 10192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73ff95eb815c438e9f94ddf171b78ea4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73ff95eb815c438e9f94ddf171b78ea4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ff95eb815c438e9f94ddf171b78ea4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35079 } } }
I20260927 03:28:51.596908 10192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73ff95eb815c438e9f94ddf171b78ea4 [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:51.597083 10193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73ff95eb815c438e9f94ddf171b78ea4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73ff95eb815c438e9f94ddf171b78ea4. Latest consensus state: current_term: 1 leader_uuid: "73ff95eb815c438e9f94ddf171b78ea4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ff95eb815c438e9f94ddf171b78ea4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35079 } } }
I20260927 03:28:51.597141 10193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73ff95eb815c438e9f94ddf171b78ea4 [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:51.597265 10207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:28:51.597893 10207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:28:51.599200 10205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:51.596891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57576 (local address 127.0.98.62:35079)
0927 03:28:51.597164 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:51.597173 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:51.597389 (+ 216us) server_negotiation.cc:407] Connection header received
0927 03:28:51.597560 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:51.597567 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:51.597616 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:51.597764 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:51.598802 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:51.598806 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:51.598806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:28:51.598809 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:51.598855 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:28:51.598866 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:51.598928 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:51.599020 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 03:28:51.599067 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":61,"threads_started":1}
I20260927 03:28:51.604547 10207 catalog_manager.cc:1347] Generated new cluster ID: 644f750224f8484c832f42137ad5fd77
I20260927 03:28:51.604631 10207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:28:51.614025 10207 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:28:51.614235 10207 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:28:51.620041 10207 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73ff95eb815c438e9f94ddf171b78ea4: Generated new TSK 0
I20260927 03:28:51.620265 10207 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:28:51.623271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35079
--builtin_ntp_servers=127.0.98.20:46495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 03:28:51.703704 10212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:51.703909 10212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:51.703933 10212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:51.705097 10212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:51.705137 10212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:51.705164 10212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:28:51.706497 10212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:51.706739 10212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:51.707043 10212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:51.712069 10217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:51.712083 10218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:51.712085 10220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:51.712682 10212 server_base.cc:1034] running on GCE node
I20260927 03:28:51.712935 10212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:51.713188 10212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:51.714356 10212 hybrid_clock.cc:648] HybridClock initialized: now 1790479731714334 us; error 40 us; skew 500 ppm
I20260927 03:28:51.715694 10212 webserver.cc:460] Webserver started at http://127.0.98.1:42719/ using document root <none> and password file <none>
I20260927 03:28:51.715950 10212 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:51.715996 10212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:51.716120 10212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:51.717159 10212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/data/instance:
uuid: "1f545b3c0b0340fca5ebb62d0a5a5abf"
format_stamp: "Formatted at 2026-09-27 03:28:51 on dist-test-slave-j0hf"
server_key: "dfb3a6c5a75db2945a4766dff406cea9"
server_key_iv: "f80ca3e31dab61f5a4e83a5e9156aa0f"
server_key_version: "encryptionkey@0"
I20260927 03:28:51.717516 10212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1f545b3c0b0340fca5ebb62d0a5a5abf"
format_stamp: "Formatted at 2026-09-27 03:28:51 on dist-test-slave-j0hf"
server_key: "dfb3a6c5a75db2945a4766dff406cea9"
server_key_iv: "f80ca3e31dab61f5a4e83a5e9156aa0f"
server_key_version: "encryptionkey@0"
I20260927 03:28:51.718813 10212 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:28:51.719605 10226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:51.719784 10212 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:51.719872 10212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "1f545b3c0b0340fca5ebb62d0a5a5abf"
format_stamp: "Formatted at 2026-09-27 03:28:51 on dist-test-slave-j0hf"
server_key: "dfb3a6c5a75db2945a4766dff406cea9"
server_key_iv: "f80ca3e31dab61f5a4e83a5e9156aa0f"
server_key_version: "encryptionkey@0"
I20260927 03:28:51.719961 10212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:51.723040 10212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:51.723307 10212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:51.723470 10212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:51.723690 10212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:51.723996 10212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:51.724056 10212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:51.724095 10212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:51.724123 10212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:51.729372 10212 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39551
I20260927 03:28:51.729444 10339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39551 every 8 connection(s)
I20260927 03:28:51.729763 10212 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:28:51.732741 10205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:51.730505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54195 (local address 127.0.98.62:35079)
0927 03:28:51.730548 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:51.730551 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:51.730766 (+ 215us) server_negotiation.cc:407] Connection header received
0927 03:28:51.731283 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:51.731284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:51.731290 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:51.731353 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:51.732477 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:51.732479 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:51.732479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:28:51.732484 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:51.732500 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:28:51.732504 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:51.732564 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:51.732649 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 03:28:51.732666 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:28:51.732920 10342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:51.730472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35079 (local address 127.0.98.1:54195)
0927 03:28:51.730639 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:51.730672 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:51.731155 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:51.731453 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:51.731457 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:51.731467 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:51.732325 (+ 858us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:28:51.732326 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:51.732352 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:28:51.732378 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:51.732564 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:51.732569 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:51.732572 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:28:51.732647 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 03:28:51.732708 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":67,"threads_started":1}
I20260927 03:28:51.733382 10340 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35079
I20260927 03:28:51.733454 10340 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:51.733621 10340 heartbeater.cc:511] Master 127.0.98.62:35079 requested a full tablet report, sending...
I20260927 03:28:51.734071 10152 ts_manager.cc:194] Registered new tserver with Master: 1f545b3c0b0340fca5ebb62d0a5a5abf (127.0.98.1:39551)
I20260927 03:28:51.738147 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 10212
I20260927 03:28:51.738224 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:28:51.738443 392 external_mini_cluster.cc:1428] Setting key f5998cef8d7798be706d4cf5de2ce483
I20260927 03:28:51.739331 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35079
--builtin_ntp_servers=127.0.98.20:46495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 03:28:51.818907 10343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:51.819101 10343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:51.819123 10343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:51.820212 10343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:51.820252 10343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:51.820278 10343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:28:51.821506 10343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:51.821681 10343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:51.821897 10343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:51.826920 10351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:51.827013 10349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:51.826951 10348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:51.827168 10343 server_base.cc:1034] running on GCE node
I20260927 03:28:51.827455 10343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:51.827713 10343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:51.828867 10343 hybrid_clock.cc:648] HybridClock initialized: now 1790479731828848 us; error 35 us; skew 500 ppm
I20260927 03:28:51.830334 10343 webserver.cc:460] Webserver started at http://127.0.98.2:45287/ using document root <none> and password file <none>
I20260927 03:28:51.830590 10343 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:51.830662 10343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:51.830818 10343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:51.831969 10343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/data/instance:
uuid: "9d5f8a46c7394101bf17a2df28bdff55"
format_stamp: "Formatted at 2026-09-27 03:28:51 on dist-test-slave-j0hf"
server_key: "63db6cf8a0b9a1962896fbe7eb6d904a"
server_key_iv: "b2ad83e20747840dbcf8efc31fba4746"
server_key_version: "encryptionkey@0"
I20260927 03:28:51.832337 10343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9d5f8a46c7394101bf17a2df28bdff55"
format_stamp: "Formatted at 2026-09-27 03:28:51 on dist-test-slave-j0hf"
server_key: "63db6cf8a0b9a1962896fbe7eb6d904a"
server_key_iv: "b2ad83e20747840dbcf8efc31fba4746"
server_key_version: "encryptionkey@0"
I20260927 03:28:51.833627 10343 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:28:51.834307 10357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:51.834472 10343 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:51.834569 10343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "9d5f8a46c7394101bf17a2df28bdff55"
format_stamp: "Formatted at 2026-09-27 03:28:51 on dist-test-slave-j0hf"
server_key: "63db6cf8a0b9a1962896fbe7eb6d904a"
server_key_iv: "b2ad83e20747840dbcf8efc31fba4746"
server_key_version: "encryptionkey@0"
I20260927 03:28:51.834638 10343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:51.837999 10343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:51.838248 10343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:51.838387 10343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:51.838584 10343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:51.838884 10343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:51.838935 10343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:51.838973 10343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:51.839001 10343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:51.844709 10343 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43205
I20260927 03:28:51.844749 10470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43205 every 8 connection(s)
I20260927 03:28:51.845057 10343 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:28:51.848069 10205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:51.845704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55895 (local address 127.0.98.62:35079)
0927 03:28:51.845757 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:51.845759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:51.845988 (+ 229us) server_negotiation.cc:407] Connection header received
0927 03:28:51.846509 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:51.846510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:51.846518 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:51.846575 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:51.847788 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:51.847789 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:51.847790 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:28:51.847794 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:51.847812 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:28:51.847816 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:51.847891 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:51.847983 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 03:28:51.847999 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:28:51.848284 10473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:51.845690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35079 (local address 127.0.98.2:55895)
0927 03:28:51.845867 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:51.845898 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:51.846389 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:51.846646 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:51.846650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:51.846660 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:51.847631 (+ 971us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:28:51.847631 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:51.847659 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:28:51.847688 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:51.847887 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:51.847893 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:51.847896 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:28:51.847996 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 03:28:51.848057 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
I20260927 03:28:51.848727 10471 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35079
I20260927 03:28:51.848834 10471 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:51.848994 10471 heartbeater.cc:511] Master 127.0.98.62:35079 requested a full tablet report, sending...
I20260927 03:28:51.849351 10152 ts_manager.cc:194] Registered new tserver with Master: 9d5f8a46c7394101bf17a2df28bdff55 (127.0.98.2:43205)
I20260927 03:28:51.853778 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 10343
I20260927 03:28:51.853899 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:28:51.854082 392 external_mini_cluster.cc:1428] Setting key 49f146d28a938bbc02bcd1cdc147ba60
I20260927 03:28:51.854964 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35079
--builtin_ntp_servers=127.0.98.20:46495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 03:28:51.938673 10474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:51.938870 10474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:51.938892 10474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:51.940100 10474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:51.940140 10474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:51.940164 10474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:28:51.941488 10474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:51.941712 10474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:51.942013 10474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:51.946894 10479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:51.947053 10480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:51.946906 10482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:51.947357 10474 server_base.cc:1034] running on GCE node
I20260927 03:28:51.947599 10474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:51.947817 10474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:51.948922 10474 hybrid_clock.cc:648] HybridClock initialized: now 1790479731948905 us; error 34 us; skew 500 ppm
I20260927 03:28:51.950213 10474 webserver.cc:460] Webserver started at http://127.0.98.3:43507/ using document root <none> and password file <none>
I20260927 03:28:51.950461 10474 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:51.950510 10474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:51.950659 10474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:51.951834 10474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/data/instance:
uuid: "ce6985fda29343f692b1bcf18829e4f4"
format_stamp: "Formatted at 2026-09-27 03:28:51 on dist-test-slave-j0hf"
server_key: "ae15b998d73d0ebb289df3c840883c15"
server_key_iv: "0a4cefaad6bf295c739fc102a732fa05"
server_key_version: "encryptionkey@0"
I20260927 03:28:51.952190 10474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ce6985fda29343f692b1bcf18829e4f4"
format_stamp: "Formatted at 2026-09-27 03:28:51 on dist-test-slave-j0hf"
server_key: "ae15b998d73d0ebb289df3c840883c15"
server_key_iv: "0a4cefaad6bf295c739fc102a732fa05"
server_key_version: "encryptionkey@0"
I20260927 03:28:51.953436 10474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 03:28:51.954128 10488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:51.954319 10474 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:51.954404 10474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "ce6985fda29343f692b1bcf18829e4f4"
format_stamp: "Formatted at 2026-09-27 03:28:51 on dist-test-slave-j0hf"
server_key: "ae15b998d73d0ebb289df3c840883c15"
server_key_iv: "0a4cefaad6bf295c739fc102a732fa05"
server_key_version: "encryptionkey@0"
I20260927 03:28:51.954473 10474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:51.957790 10474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:51.958047 10474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:51.958187 10474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:51.958388 10474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:51.958745 10474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:51.958786 10474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:51.958806 10474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:51.958848 10474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:51.964432 10474 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45277
I20260927 03:28:51.964489 10601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45277 every 8 connection(s)
I20260927 03:28:51.964785 10474 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:28:51.967654 10205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:51.965491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53443 (local address 127.0.98.62:35079)
0927 03:28:51.965554 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:51.965557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:51.965795 (+ 238us) server_negotiation.cc:407] Connection header received
0927 03:28:51.966228 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:51.966229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:51.966234 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:51.966288 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:51.967409 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:51.967410 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:51.967410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:28:51.967413 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:51.967455 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:28:51.967460 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:51.967523 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:51.967589 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 03:28:51.967602 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 03:28:51.967852 10604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:51.965479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35079 (local address 127.0.98.3:53443)
0927 03:28:51.965668 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:51.965700 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:51.966136 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:51.966340 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:51.966343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:51.966351 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:51.967284 (+ 933us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:28:51.967284 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:51.967311 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:28:51.967338 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:51.967523 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:51.967526 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:51.967528 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:28:51.967585 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 03:28:51.967632 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":106,"threads_started":1}
I20260927 03:28:51.968319 10602 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35079
I20260927 03:28:51.968391 10602 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:51.968493 10602 heartbeater.cc:511] Master 127.0.98.62:35079 requested a full tablet report, sending...
I20260927 03:28:51.968791 10152 ts_manager.cc:194] Registered new tserver with Master: ce6985fda29343f692b1bcf18829e4f4 (127.0.98.3:45277)
I20260927 03:28:51.969265 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 10474
I20260927 03:28:51.969367 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790479675865103-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:28:51.969549 392 external_mini_cluster.cc:1428] Setting key 843f93b2fd17249102b7d9e26aa2163f
I20260927 03:28:51.970356 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 03:28:51.972083 10205 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:51.971624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57580 (local address 127.0.98.62:35079)
0927 03:28:51.971658 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:51.971660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:51.971786 (+ 126us) server_negotiation.cc:407] Connection header received
0927 03:28:51.971863 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:51.971864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:51.971868 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:51.971923 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:51.972028 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57580: BlockingRecv error: recv got EOF from 127.0.0.1:57580 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 03:28:51.972223 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 10212
I20260927 03:28:51.976892 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 10343
I20260927 03:28:51.981318 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 10474
I20260927 03:28:51.985347 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 10121
2026-09-27T03:28:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (530 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:28:52 dist-test-slave-j0hf krb5kdc[10612](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:28:52 dist-test-slave-j0hf krb5kdc[10612](info): set up 2 sockets
Sep 27 03:28:52 dist-test-slave-j0hf krb5kdc[10612](info): commencing operation
krb5kdc: starting...
W20260927 03:28:53.008471 392 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:28:53 dist-test-slave-j0hf krb5kdc[10612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:28:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:28:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:28:53.123992 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36829
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:53.204455 10625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:53.204654 10625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:53.204677 10625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:53.205814 10625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:28:53.205848 10625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:53.205862 10625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:53.205875 10625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:28:53.205886 10625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:28:53.207329 10625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:53.207580 10625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:53.207767 10625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:53.212339 10633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:53.212388 10630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:53.212397 10631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:53.212486 10625 server_base.cc:1034] running on GCE node
I20260927 03:28:53.212990 10625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:53.213290 10625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:53.214432 10625 hybrid_clock.cc:648] HybridClock initialized: now 1790479733214417 us; error 38 us; skew 500 ppm
Sep 27 03:28:53 dist-test-slave-j0hf krb5kdc[10612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:53.216964 10625 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:28:53.217376 10625 webserver.cc:460] Webserver started at http://127.0.98.62:36607/ using document root <none> and password file <none>
I20260927 03:28:53.217614 10625 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:53.217679 10625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:53.217819 10625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:53.218852 10625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "ef49c6899b81429fb0e64b151465b1d6"
format_stamp: "Formatted at 2026-09-27 03:28:53 on dist-test-slave-j0hf"
server_key: "dbb66de569afa5de422ebbe96f25d543"
server_key_iv: "f34b654be91c2bd1991d74c880bb53c1"
server_key_version: "encryptionkey@0"
I20260927 03:28:53.219200 10625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "ef49c6899b81429fb0e64b151465b1d6"
format_stamp: "Formatted at 2026-09-27 03:28:53 on dist-test-slave-j0hf"
server_key: "dbb66de569afa5de422ebbe96f25d543"
server_key_iv: "f34b654be91c2bd1991d74c880bb53c1"
server_key_version: "encryptionkey@0"
I20260927 03:28:53.220546 10625 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 03:28:53.221244 10640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:53.221392 10625 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:53.221483 10625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "ef49c6899b81429fb0e64b151465b1d6"
format_stamp: "Formatted at 2026-09-27 03:28:53 on dist-test-slave-j0hf"
server_key: "dbb66de569afa5de422ebbe96f25d543"
server_key_iv: "f34b654be91c2bd1991d74c880bb53c1"
server_key_version: "encryptionkey@0"
I20260927 03:28:53.221561 10625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:53.242660 10625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:53.244653 10625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:53.244828 10625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:53.248742 10625 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36829
I20260927 03:28:53.248781 10692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36829 every 8 connection(s)
I20260927 03:28:53.249126 10625 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:28:53.250196 10693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:53.252454 10693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef49c6899b81429fb0e64b151465b1d6: Bootstrap starting.
I20260927 03:28:53.252960 10693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef49c6899b81429fb0e64b151465b1d6: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:53.253234 10693 log.cc:824] T 00000000000000000000000000000000 P ef49c6899b81429fb0e64b151465b1d6: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:53.253866 10693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef49c6899b81429fb0e64b151465b1d6: No bootstrap required, opened a new log
I20260927 03:28:53.255398 10693 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef49c6899b81429fb0e64b151465b1d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef49c6899b81429fb0e64b151465b1d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36829 } }
I20260927 03:28:53.255606 10693 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef49c6899b81429fb0e64b151465b1d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:53.255635 10693 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef49c6899b81429fb0e64b151465b1d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef49c6899b81429fb0e64b151465b1d6, State: Initialized, Role: FOLLOWER
I20260927 03:28:53.255766 10693 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef49c6899b81429fb0e64b151465b1d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef49c6899b81429fb0e64b151465b1d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36829 } }
I20260927 03:28:53.255857 10693 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef49c6899b81429fb0e64b151465b1d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:28:53.255896 10693 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef49c6899b81429fb0e64b151465b1d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:28:53.255930 10693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef49c6899b81429fb0e64b151465b1d6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:53.256662 10693 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef49c6899b81429fb0e64b151465b1d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef49c6899b81429fb0e64b151465b1d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36829 } }
I20260927 03:28:53.256740 10693 leader_election.cc:302] T 00000000000000000000000000000000 P ef49c6899b81429fb0e64b151465b1d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef49c6899b81429fb0e64b151465b1d6; no voters:
I20260927 03:28:53.256850 10693 leader_election.cc:288] T 00000000000000000000000000000000 P ef49c6899b81429fb0e64b151465b1d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:28:53.256904 10696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef49c6899b81429fb0e64b151465b1d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:53.257043 10696 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef49c6899b81429fb0e64b151465b1d6 [term 1 LEADER]: Becoming Leader. State: Replica: ef49c6899b81429fb0e64b151465b1d6, State: Running, Role: LEADER
I20260927 03:28:53.257089 10693 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef49c6899b81429fb0e64b151465b1d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:28:53.257125 10696 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef49c6899b81429fb0e64b151465b1d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef49c6899b81429fb0e64b151465b1d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36829 } }
I20260927 03:28:53.257433 10697 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef49c6899b81429fb0e64b151465b1d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef49c6899b81429fb0e64b151465b1d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef49c6899b81429fb0e64b151465b1d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36829 } } }
I20260927 03:28:53.257428 10698 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef49c6899b81429fb0e64b151465b1d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef49c6899b81429fb0e64b151465b1d6. Latest consensus state: current_term: 1 leader_uuid: "ef49c6899b81429fb0e64b151465b1d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef49c6899b81429fb0e64b151465b1d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36829 } } }
I20260927 03:28:53.257515 10697 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef49c6899b81429fb0e64b151465b1d6 [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:53.257524 10698 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef49c6899b81429fb0e64b151465b1d6 [sys.catalog]: This master's current role is: LEADER
I20260927 03:28:53.257845 10702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:28:53.258387 10702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:28:53.258800 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 10625
I20260927 03:28:53.258926 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:28:53.259133 392 external_mini_cluster.cc:1428] Setting key f19c47cf43858ff4680491c3450fff69
I20260927 03:28:53.259930 10702 catalog_manager.cc:1347] Generated new cluster ID: 686ccca399a94ed18e51dce1015f4659
I20260927 03:28:53.259981 10702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:28:53 dist-test-slave-j0hf krb5kdc[10612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:53.276332 10716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:53.259840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53616 (local address 127.0.98.62:36829)
0927 03:28:53.267564 (+ 7724us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:53.267573 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:53.267606 (+ 33us) server_negotiation.cc:407] Connection header received
0927 03:28:53.267720 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:53.267726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:53.267773 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:53.267931 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:53.268562 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:53.269486 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:53.270217 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:53.270640 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:53.270759 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:53.273155 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:53.273164 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:53.273164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:53.273176 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:53.274843 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:53.275196 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:53.275199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:53.275199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:53.275289 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:53.275489 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:53.275490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:53.275490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:53.275730 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:53.275879 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:53.275996 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 03:28:53.276139 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7665,"thread_start_us":64,"threads_started":1}
I20260927 03:28:53.306166 10702 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:28:53.306793 10702 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:28:53.318821 10702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef49c6899b81429fb0e64b151465b1d6: Generated new TSK 0
I20260927 03:28:53.319048 10702 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:28:53.350325 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36829
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:53.457885 10719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:53.458084 10719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:53.458107 10719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:53.459309 10719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:53.459348 10719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:53.459374 10719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:28:53.460842 10719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:53.461104 10719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:53.461328 10719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:53.466126 10727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:53.466115 10724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:53.466114 10725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:53.466432 10719 server_base.cc:1034] running on GCE node
I20260927 03:28:53.466640 10719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:53.466899 10719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:53.468070 10719 hybrid_clock.cc:648] HybridClock initialized: now 1790479733468059 us; error 48 us; skew 500 ppm
Sep 27 03:28:53 dist-test-slave-j0hf krb5kdc[10612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:53.470559 10719 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:28:53.471011 10719 webserver.cc:460] Webserver started at http://127.0.98.1:37855/ using document root <none> and password file <none>
I20260927 03:28:53.471230 10719 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:53.471295 10719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:53.471491 10719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:53.472527 10719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-0/data/instance:
uuid: "66501d03b6f54ca6aa1016e82bd65358"
format_stamp: "Formatted at 2026-09-27 03:28:53 on dist-test-slave-j0hf"
server_key: "2dd6bba2f977d99da17d1881ca496d7d"
server_key_iv: "9fc0fbe30e0e69779e367623c1ba5e3b"
server_key_version: "encryptionkey@0"
I20260927 03:28:53.472882 10719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-0/wal/instance:
uuid: "66501d03b6f54ca6aa1016e82bd65358"
format_stamp: "Formatted at 2026-09-27 03:28:53 on dist-test-slave-j0hf"
server_key: "2dd6bba2f977d99da17d1881ca496d7d"
server_key_iv: "9fc0fbe30e0e69779e367623c1ba5e3b"
server_key_version: "encryptionkey@0"
I20260927 03:28:53.474215 10719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:28:53.474967 10734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:53.475133 10719 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:53.475226 10719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "66501d03b6f54ca6aa1016e82bd65358"
format_stamp: "Formatted at 2026-09-27 03:28:53 on dist-test-slave-j0hf"
server_key: "2dd6bba2f977d99da17d1881ca496d7d"
server_key_iv: "9fc0fbe30e0e69779e367623c1ba5e3b"
server_key_version: "encryptionkey@0"
I20260927 03:28:53.475306 10719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:53.484727 10719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:53.486780 10719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:53.486897 10719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:53.487082 10719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:53.487332 10719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:53.487362 10719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:53.487385 10719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:53.487397 10719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:53.493075 10719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39795
I20260927 03:28:53.493186 10847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39795 every 8 connection(s)
I20260927 03:28:53.493494 10719 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:28:53.495774 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 10719
I20260927 03:28:53.495885 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:28:53.496091 392 external_mini_cluster.cc:1428] Setting key 07fc9188d35df3b78b5732abe0634757
Sep 27 03:28:53 dist-test-slave-j0hf krb5kdc[10612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:53.503509 10716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:53.494239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42393 (local address 127.0.98.62:36829)
0927 03:28:53.494310 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:53.494312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:53.494506 (+ 194us) server_negotiation.cc:407] Connection header received
0927 03:28:53.494999 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:53.495001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:53.495011 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:53.495070 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:53.496122 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:53.496821 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:53.497906 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:53.498170 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:53.498222 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:53.500663 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:53.500664 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:53.500665 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:53.500672 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:53.502105 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:53.502519 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:53.502520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:53.502521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:53.502578 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:53.502832 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:53.502833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:53.502833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:53.502945 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:53.503031 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:53.503302 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 03:28:53.503390 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:28:53.503705 10850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:53.494202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36829 (local address 127.0.98.1:42393)
0927 03:28:53.494376 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:53.494418 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:53.494902 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:53.495131 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:53.495135 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:53.495777 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:53.496034 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:53.496039 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:53.496926 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:53.496926 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:53.497766 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:53.497768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:53.498254 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:53.498255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:53.498404 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:53.498922 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:53.498932 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:53.500536 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:53.502205 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:53.502213 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:53.502215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:53.502438 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:53.502657 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:53.502658 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:53.502659 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:53.502768 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:53.503024 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:53.503030 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:53.503175 (+ 145us) client_negotiation.cc:769] Sending connection context
0927 03:28:53.503281 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 03:28:53.503460 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 03:28:53.504320 10848 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36829
I20260927 03:28:53.504417 10848 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:53.504560 10848 heartbeater.cc:511] Master 127.0.98.62:36829 requested a full tablet report, sending...
I20260927 03:28:53.505136 10657 ts_manager.cc:194] Registered new tserver with Master: 66501d03b6f54ca6aa1016e82bd65358 (127.0.98.1:39795)
I20260927 03:28:53.505882 10657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:28:53.521796 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36829
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:53.604842 10853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:53.605034 10853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:53.605057 10853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:53.606186 10853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:53.606240 10853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:53.606267 10853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:28:53.607698 10853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:53.607931 10853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:53.608155 10853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:53.613111 10859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:53.613111 10858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:53.613144 10853 server_base.cc:1034] running on GCE node
W20260927 03:28:53.613129 10861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:53.613518 10853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:53.613758 10853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:53.614923 10853 hybrid_clock.cc:648] HybridClock initialized: now 1790479733614908 us; error 45 us; skew 500 ppm
Sep 27 03:28:53 dist-test-slave-j0hf krb5kdc[10612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:53.617385 10853 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:28:53.617838 10853 webserver.cc:460] Webserver started at http://127.0.98.2:45007/ using document root <none> and password file <none>
I20260927 03:28:53.618079 10853 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:53.618144 10853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:53.618283 10853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:53.619313 10853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-1/data/instance:
uuid: "24204c579ecb4926bb8e8d07ed4d68f2"
format_stamp: "Formatted at 2026-09-27 03:28:53 on dist-test-slave-j0hf"
server_key: "98d0bddb1e00e32d519f509c180fd460"
server_key_iv: "f19f429b87513d8b4cb7276bd7756aae"
server_key_version: "encryptionkey@0"
I20260927 03:28:53.619711 10853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-1/wal/instance:
uuid: "24204c579ecb4926bb8e8d07ed4d68f2"
format_stamp: "Formatted at 2026-09-27 03:28:53 on dist-test-slave-j0hf"
server_key: "98d0bddb1e00e32d519f509c180fd460"
server_key_iv: "f19f429b87513d8b4cb7276bd7756aae"
server_key_version: "encryptionkey@0"
I20260927 03:28:53.620983 10853 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:28:53.621678 10868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:53.621842 10853 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:28:53.621933 10853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "24204c579ecb4926bb8e8d07ed4d68f2"
format_stamp: "Formatted at 2026-09-27 03:28:53 on dist-test-slave-j0hf"
server_key: "98d0bddb1e00e32d519f509c180fd460"
server_key_iv: "f19f429b87513d8b4cb7276bd7756aae"
server_key_version: "encryptionkey@0"
I20260927 03:28:53.622010 10853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:53.658489 10853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:53.660853 10853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:53.661017 10853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:53.661257 10853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:53.661607 10853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:53.661660 10853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:53.661702 10853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:53.661728 10853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:53.667536 10853 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39673
I20260927 03:28:53.667577 10981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39673 every 8 connection(s)
I20260927 03:28:53.667902 10853 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:28:53 dist-test-slave-j0hf krb5kdc[10612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:53.676527 10716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:53.668613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43377 (local address 127.0.98.62:36829)
0927 03:28:53.668658 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:53.668660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:53.668850 (+ 190us) server_negotiation.cc:407] Connection header received
0927 03:28:53.669338 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:53.669339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:53.669348 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:53.669414 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:53.670327 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:53.670816 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:53.671870 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:53.672054 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:53.672103 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:53.674201 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:53.674202 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:53.674203 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:53.674209 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:53.675392 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:53.675789 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:53.675789 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:53.675790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:53.675841 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:53.676039 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:53.676039 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:53.676039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:53.676135 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:53.676187 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:53.676405 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 03:28:53.676463 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:28:53.676688 10984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:53.668579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36829 (local address 127.0.98.2:43377)
0927 03:28:53.668732 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:53.668775 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:53.669258 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:53.669470 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:53.669473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:53.670032 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:53.670231 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:53.670245 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:53.670900 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:53.670901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:53.671794 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:53.671797 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:53.672121 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:53.672121 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:53.672216 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:53.672756 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:53.672764 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:53.674119 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:53.675516 (+ 1397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:53.675522 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:53.675524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:53.675727 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:53.675895 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:53.675895 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:53.675895 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:53.675986 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:53.676184 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:53.676188 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:53.676304 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 03:28:53.676384 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 03:28:53.676493 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260927 03:28:53.677161 10982 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36829
I20260927 03:28:53.677263 10982 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:53.677444 10982 heartbeater.cc:511] Master 127.0.98.62:36829 requested a full tablet report, sending...
I20260927 03:28:53.677621 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 10853
I20260927 03:28:53.677732 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:28:53.677874 10657 ts_manager.cc:194] Registered new tserver with Master: 24204c579ecb4926bb8e8d07ed4d68f2 (127.0.98.2:39673)
I20260927 03:28:53.677994 392 external_mini_cluster.cc:1428] Setting key b2fa97f1342ac9077bb57ab63225fe4a
I20260927 03:28:53.678357 10657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43377
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:28:53.701915 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36829
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:28:53.782124 10987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:28:53.782348 10987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:28:53.782387 10987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:28:53.783572 10987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:28:53.783627 10987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:28:53.783669 10987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:28:53.785123 10987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:28:53.785380 10987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:28:53.785602 10987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:28:53.790378 10992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:53.790484 10993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:28:53.790382 10995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:28:53.790723 10987 server_base.cc:1034] running on GCE node
I20260927 03:28:53.790930 10987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:28:53.791193 10987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:28:53.792338 10987 hybrid_clock.cc:648] HybridClock initialized: now 1790479733792320 us; error 36 us; skew 500 ppm
Sep 27 03:28:53 dist-test-slave-j0hf krb5kdc[10612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:28:53.794655 10987 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:28:53.795089 10987 webserver.cc:460] Webserver started at http://127.0.98.3:33189/ using document root <none> and password file <none>
I20260927 03:28:53.795313 10987 fs_manager.cc:359] Metadata directory not provided
I20260927 03:28:53.795377 10987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:28:53.795543 10987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:28:53.796553 10987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-2/data/instance:
uuid: "c27391f44d0049c18be482bc48a2cf95"
format_stamp: "Formatted at 2026-09-27 03:28:53 on dist-test-slave-j0hf"
server_key: "17a5137377b5b432946d9c3d9aee53e4"
server_key_iv: "66dbac717750e5c78276e19c7638e5d9"
server_key_version: "encryptionkey@0"
I20260927 03:28:53.796900 10987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c27391f44d0049c18be482bc48a2cf95"
format_stamp: "Formatted at 2026-09-27 03:28:53 on dist-test-slave-j0hf"
server_key: "17a5137377b5b432946d9c3d9aee53e4"
server_key_iv: "66dbac717750e5c78276e19c7638e5d9"
server_key_version: "encryptionkey@0"
I20260927 03:28:53.798146 10987 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:28:53.798913 11002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:53.799072 10987 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:28:53.799165 10987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "c27391f44d0049c18be482bc48a2cf95"
format_stamp: "Formatted at 2026-09-27 03:28:53 on dist-test-slave-j0hf"
server_key: "17a5137377b5b432946d9c3d9aee53e4"
server_key_iv: "66dbac717750e5c78276e19c7638e5d9"
server_key_version: "encryptionkey@0"
I20260927 03:28:53.799238 10987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:28:53.811498 10987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:28:53.813316 10987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:28:53.813474 10987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:28:53.813678 10987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:28:53.813969 10987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:28:53.814018 10987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:53.814059 10987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:28:53.814085 10987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:28:53.820129 10987 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41085
I20260927 03:28:53.820170 11115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41085 every 8 connection(s)
I20260927 03:28:53.820470 10987 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:28:53 dist-test-slave-j0hf krb5kdc[10612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:28:53.826555 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 10987
I20260927 03:28:53.826642 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:28:53.826831 392 external_mini_cluster.cc:1428] Setting key 3d8f39595d9f9e18be47b617b0c479ce
I20260927 03:28:53.829576 10716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:53.821113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58485 (local address 127.0.98.62:36829)
0927 03:28:53.821177 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:53.821179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:53.821381 (+ 202us) server_negotiation.cc:407] Connection header received
0927 03:28:53.821831 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:53.821832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:53.821842 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:53.821893 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:53.822744 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:53.823242 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:53.824306 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:53.824647 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:53.824696 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:53.826952 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:53.826953 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:53.826954 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:53.826962 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:53.828467 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:53.828815 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:53.828816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:53.828816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:53.828864 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:53.829081 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:53.829082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:53.829082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:53.829191 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:53.829242 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:53.829458 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 03:28:53.829513 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:28:53.829813 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:53.821107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36829 (local address 127.0.98.3:58485)
0927 03:28:53.821268 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:53.821301 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:53.821753 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:53.821943 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:53.821946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:53.822482 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:28:53.822661 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:53.822665 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:53.823316 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:53.823317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:53.824240 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:53.824242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:53.824712 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:53.824713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:53.824822 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:53.825318 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:28:53.825327 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:28:53.826825 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:28:53.828534 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:53.828540 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:53.828542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:53.828752 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:53.828933 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:28:53.828934 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:28:53.828934 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:28:53.829025 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:28:53.829241 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:28:53.829245 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:28:53.829361 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 03:28:53.829445 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 03:28:53.829556 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
I20260927 03:28:53.830263 11116 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36829
I20260927 03:28:53.830338 11116 heartbeater.cc:464] Registering TS with master...
I20260927 03:28:53.830476 11116 heartbeater.cc:511] Master 127.0.98.62:36829 requested a full tablet report, sending...
I20260927 03:28:53.830812 10656 ts_manager.cc:194] Registered new tserver with Master: c27391f44d0049c18be482bc48a2cf95 (127.0.98.3:41085)
I20260927 03:28:53.831233 10656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58485
I20260927 03:28:53.838554 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:28:53.845295 10716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:53.839926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53618 (local address 127.0.98.62:36829)
0927 03:28:53.839963 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:53.839965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:53.840068 (+ 103us) server_negotiation.cc:407] Connection header received
0927 03:28:53.840106 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:53.840107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:53.840114 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:53.840162 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:28:53.840651 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:53.841132 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:53.841791 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:53.841961 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:53.842006 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:53.843146 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:28:53.843147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:28:53.843147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:28:53.843152 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:28:53.844432 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:53.844717 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:53.844718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:53.844718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:53.844763 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:28:53.844899 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:28:53.844899 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:28:53.844900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:28:53.845016 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:28:53.845083 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:53.845168 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 03:28:53.845219 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 03:28:53.846755 10656 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:28:53.848316 10656 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:28:53.854605 11129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:53.850537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39673 (local address 127.0.0.1:41132)
0927 03:28:53.850784 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:53.850828 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:53.850892 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:53.851186 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:53.851189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:53.851191 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:53.851268 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:53.851270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:53.852286 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:53.852287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:53.853524 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:53.853527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:53.854326 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:53.854327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:53.854402 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:53.854415 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 03:28:53.854484 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 03:28:53.854534 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":7,"thread_start_us":42,"threads_started":1}
I20260927 03:28:53.854617 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:53.850903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41132 (local address 127.0.98.2:39673)
0927 03:28:53.851044 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:53.851047 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:53.851054 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:28:53.851084 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:53.851086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:53.851098 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:53.851184 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:53.851353 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:53.852165 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:53.853594 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:53.854245 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:53.854328 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:53.854388 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:53.854496 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 03:28:53.854536 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
I20260927 03:28:53.854984 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:53.850548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41085 (local address 127.0.0.1:60228)
0927 03:28:53.851190 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:53.851199 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:53.851447 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:53.851708 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:53.851709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:53.851711 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:53.851756 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:53.851758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:53.852924 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:53.852925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:53.853899 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:53.853902 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:53.854748 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:53.854749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:53.854814 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:53.854818 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:28:53.854879 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 03:28:53.854914 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":78,"threads_started":1}
I20260927 03:28:53.855006 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:53.850744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60228 (local address 127.0.98.3:41085)
0927 03:28:53.850922 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:53.850926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:53.851400 (+ 474us) server_negotiation.cc:407] Connection header received
0927 03:28:53.851543 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:53.851545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:53.851561 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:53.851627 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:53.851832 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:53.852834 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:53.853982 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:53.854668 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:53.854745 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:53.854803 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:53.854890 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 03:28:53.854926 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":73,"threads_started":1}
I20260927 03:28:53.855392 11127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:53.850538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39795 (local address 127.0.0.1:42104)
0927 03:28:53.851030 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:53.851038 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:53.851108 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:53.851280 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:53.851281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:53.851283 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:53.851327 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:53.851328 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:53.852697 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:53.852698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:53.853712 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:53.853714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:53.854640 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:53.854641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:53.855019 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:53.855249 (+ 230us) client_negotiation.cc:769] Sending connection context
0927 03:28:53.855305 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 03:28:53.855348 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":88,"threads_started":1}
I20260927 03:28:53.855383 10916 tablet_service.cc:1467] Processing CreateTablet for tablet e0f134495c3647f9b1216fcb268adebe (DEFAULT_TABLE table=test-table [id=312d01e4c38f4d8ebbaac17429ceba85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:53.855710 11130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:53.850702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42104 (local address 127.0.98.1:39795)
0927 03:28:53.850866 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:53.850871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:53.851103 (+ 232us) server_negotiation.cc:407] Connection header received
0927 03:28:53.851180 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:53.851189 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:53.851208 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:53.851287 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:53.851474 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:53.852580 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:53.853801 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:53.854485 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:53.854555 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:53.854606 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:53.855583 (+ 977us) server_negotiation.cc:299] Negotiation successful
0927 03:28:53.855632 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
I20260927 03:28:53.855676 11050 tablet_service.cc:1467] Processing CreateTablet for tablet e0f134495c3647f9b1216fcb268adebe (DEFAULT_TABLE table=test-table [id=312d01e4c38f4d8ebbaac17429ceba85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:53.855783 10916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f134495c3647f9b1216fcb268adebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:53.855958 11050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f134495c3647f9b1216fcb268adebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:53.856254 10782 tablet_service.cc:1467] Processing CreateTablet for tablet e0f134495c3647f9b1216fcb268adebe (DEFAULT_TABLE table=test-table [id=312d01e4c38f4d8ebbaac17429ceba85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:28:53.856535 10782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f134495c3647f9b1216fcb268adebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:28:53.859105 11133 tablet_bootstrap.cc:492] T e0f134495c3647f9b1216fcb268adebe P 24204c579ecb4926bb8e8d07ed4d68f2: Bootstrap starting.
I20260927 03:28:53.859462 11135 tablet_bootstrap.cc:492] T e0f134495c3647f9b1216fcb268adebe P 66501d03b6f54ca6aa1016e82bd65358: Bootstrap starting.
I20260927 03:28:53.859874 11133 tablet_bootstrap.cc:654] T e0f134495c3647f9b1216fcb268adebe P 24204c579ecb4926bb8e8d07ed4d68f2: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:53.860075 11135 tablet_bootstrap.cc:654] T e0f134495c3647f9b1216fcb268adebe P 66501d03b6f54ca6aa1016e82bd65358: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:53.860517 11133 log.cc:824] T e0f134495c3647f9b1216fcb268adebe P 24204c579ecb4926bb8e8d07ed4d68f2: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:53.860685 11135 log.cc:824] T e0f134495c3647f9b1216fcb268adebe P 66501d03b6f54ca6aa1016e82bd65358: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:53.860882 11134 tablet_bootstrap.cc:492] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95: Bootstrap starting.
I20260927 03:28:53.861467 11135 tablet_bootstrap.cc:492] T e0f134495c3647f9b1216fcb268adebe P 66501d03b6f54ca6aa1016e82bd65358: No bootstrap required, opened a new log
I20260927 03:28:53.861526 11134 tablet_bootstrap.cc:654] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95: Neither blocks nor log segments found. Creating new log.
I20260927 03:28:53.861545 11135 ts_tablet_manager.cc:1397] T e0f134495c3647f9b1216fcb268adebe P 66501d03b6f54ca6aa1016e82bd65358: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:28:53.862084 11134 log.cc:824] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95: Log is configured to *not* fsync() on all Append() calls
I20260927 03:28:53.862582 11133 tablet_bootstrap.cc:492] T e0f134495c3647f9b1216fcb268adebe P 24204c579ecb4926bb8e8d07ed4d68f2: No bootstrap required, opened a new log
I20260927 03:28:53.862653 11133 ts_tablet_manager.cc:1397] T e0f134495c3647f9b1216fcb268adebe P 24204c579ecb4926bb8e8d07ed4d68f2: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260927 03:28:53.862771 11134 tablet_bootstrap.cc:492] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95: No bootstrap required, opened a new log
I20260927 03:28:53.862854 11134 ts_tablet_manager.cc:1397] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:28:53.863634 11135 raft_consensus.cc:348] T e0f134495c3647f9b1216fcb268adebe P 66501d03b6f54ca6aa1016e82bd65358 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24204c579ecb4926bb8e8d07ed4d68f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39673 } } peers { permanent_uuid: "66501d03b6f54ca6aa1016e82bd65358" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39795 } } peers { permanent_uuid: "c27391f44d0049c18be482bc48a2cf95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41085 } }
I20260927 03:28:53.863817 11135 raft_consensus.cc:374] T e0f134495c3647f9b1216fcb268adebe P 66501d03b6f54ca6aa1016e82bd65358 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:53.863854 11135 raft_consensus.cc:729] T e0f134495c3647f9b1216fcb268adebe P 66501d03b6f54ca6aa1016e82bd65358 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66501d03b6f54ca6aa1016e82bd65358, State: Initialized, Role: FOLLOWER
I20260927 03:28:53.864003 11135 consensus_queue.cc:260] T e0f134495c3647f9b1216fcb268adebe P 66501d03b6f54ca6aa1016e82bd65358 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24204c579ecb4926bb8e8d07ed4d68f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39673 } } peers { permanent_uuid: "66501d03b6f54ca6aa1016e82bd65358" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39795 } } peers { permanent_uuid: "c27391f44d0049c18be482bc48a2cf95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41085 } }
I20260927 03:28:53.864279 11135 ts_tablet_manager.cc:1428] T e0f134495c3647f9b1216fcb268adebe P 66501d03b6f54ca6aa1016e82bd65358: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:28:53.864396 10848 heartbeater.cc:503] Master 127.0.98.62:36829 was elected leader, sending a full tablet report...
I20260927 03:28:53.864447 11134 raft_consensus.cc:348] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24204c579ecb4926bb8e8d07ed4d68f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39673 } } peers { permanent_uuid: "66501d03b6f54ca6aa1016e82bd65358" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39795 } } peers { permanent_uuid: "c27391f44d0049c18be482bc48a2cf95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41085 } }
I20260927 03:28:53.864612 11134 raft_consensus.cc:374] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:53.864652 11134 raft_consensus.cc:729] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c27391f44d0049c18be482bc48a2cf95, State: Initialized, Role: FOLLOWER
I20260927 03:28:53.864787 11134 consensus_queue.cc:260] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24204c579ecb4926bb8e8d07ed4d68f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39673 } } peers { permanent_uuid: "66501d03b6f54ca6aa1016e82bd65358" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39795 } } peers { permanent_uuid: "c27391f44d0049c18be482bc48a2cf95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41085 } }
I20260927 03:28:53.864816 11133 raft_consensus.cc:348] T e0f134495c3647f9b1216fcb268adebe P 24204c579ecb4926bb8e8d07ed4d68f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24204c579ecb4926bb8e8d07ed4d68f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39673 } } peers { permanent_uuid: "66501d03b6f54ca6aa1016e82bd65358" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39795 } } peers { permanent_uuid: "c27391f44d0049c18be482bc48a2cf95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41085 } }
I20260927 03:28:53.864969 11133 raft_consensus.cc:374] T e0f134495c3647f9b1216fcb268adebe P 24204c579ecb4926bb8e8d07ed4d68f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:28:53.865015 11133 raft_consensus.cc:729] T e0f134495c3647f9b1216fcb268adebe P 24204c579ecb4926bb8e8d07ed4d68f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24204c579ecb4926bb8e8d07ed4d68f2, State: Initialized, Role: FOLLOWER
I20260927 03:28:53.865125 11133 consensus_queue.cc:260] T e0f134495c3647f9b1216fcb268adebe P 24204c579ecb4926bb8e8d07ed4d68f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24204c579ecb4926bb8e8d07ed4d68f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39673 } } peers { permanent_uuid: "66501d03b6f54ca6aa1016e82bd65358" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39795 } } peers { permanent_uuid: "c27391f44d0049c18be482bc48a2cf95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41085 } }
I20260927 03:28:53.865243 11134 ts_tablet_manager.cc:1428] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 03:28:53.865264 11116 heartbeater.cc:503] Master 127.0.98.62:36829 was elected leader, sending a full tablet report...
I20260927 03:28:53.865402 11133 ts_tablet_manager.cc:1428] T e0f134495c3647f9b1216fcb268adebe P 24204c579ecb4926bb8e8d07ed4d68f2: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260927 03:28:53.865437 10982 heartbeater.cc:503] Master 127.0.98.62:36829 was elected leader, sending a full tablet report...
W20260927 03:28:53.918512 10983 tablet.cc:2369] T e0f134495c3647f9b1216fcb268adebe P 24204c579ecb4926bb8e8d07ed4d68f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:28:53.994258 10849 tablet.cc:2369] T e0f134495c3647f9b1216fcb268adebe P 66501d03b6f54ca6aa1016e82bd65358: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:28:54.054452 11140 raft_consensus.cc:482] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:28:54.054580 11140 raft_consensus.cc:504] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24204c579ecb4926bb8e8d07ed4d68f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39673 } } peers { permanent_uuid: "66501d03b6f54ca6aa1016e82bd65358" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39795 } } peers { permanent_uuid: "c27391f44d0049c18be482bc48a2cf95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41085 } }
I20260927 03:28:54.054899 11140 leader_election.cc:288] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24204c579ecb4926bb8e8d07ed4d68f2 (127.0.98.2:39673), 66501d03b6f54ca6aa1016e82bd65358 (127.0.98.1:39795)
I20260927 03:28:54.058881 11130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:54.055227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54001 (local address 127.0.98.1:39795)
0927 03:28:54.055287 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:54.055289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:54.055436 (+ 147us) server_negotiation.cc:407] Connection header received
0927 03:28:54.055539 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:54.055540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:54.055553 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:54.055610 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:54.055824 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:54.056533 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:54.058028 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:54.058490 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:54.058545 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:54.058554 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:54.058753 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 03:28:54.058794 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 03:28:54.058919 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:54.055157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39795 (local address 127.0.98.3:54001)
0927 03:28:54.055357 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:54.055367 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:54.055469 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:54.055670 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:54.055671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:54.055679 (+ 8us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:54.055748 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:54.055749 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:54.056639 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:54.056640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:54.057931 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:54.057935 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:54.058571 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:54.058572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:54.058667 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:54.058674 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 03:28:54.058749 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 03:28:54.058824 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":14,"thread_start_us":113,"threads_started":1}
I20260927 03:28:54.059089 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:54.055227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48385 (local address 127.0.98.2:39673)
0927 03:28:54.055277 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:54.055279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:54.055293 (+ 14us) server_negotiation.cc:407] Connection header received
0927 03:28:54.055385 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:54.055386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:54.055396 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:54.055523 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:28:54.055731 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:54.056483 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:54.058081 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:54.058609 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:54.058700 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:54.058713 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:54.058958 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 03:28:54.059014 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:28:54.059257 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:54.055146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39673 (local address 127.0.98.3:48385)
0927 03:28:54.055224 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:54.055240 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:54.055295 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:28:54.055536 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:28:54.055537 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:28:54.055539 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:28:54.055644 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:54.055647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:54.056559 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:54.056560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:54.057962 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:28:54.057966 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:28:54.058712 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:28:54.058713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:28:54.058796 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:54.058799 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:28:54.058852 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 03:28:54.058891 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":88}
I20260927 03:28:54.059315 10802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f134495c3647f9b1216fcb268adebe" candidate_uuid: "c27391f44d0049c18be482bc48a2cf95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66501d03b6f54ca6aa1016e82bd65358" is_pre_election: true
I20260927 03:28:54.059482 10802 raft_consensus.cc:2454] T e0f134495c3647f9b1216fcb268adebe P 66501d03b6f54ca6aa1016e82bd65358 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c27391f44d0049c18be482bc48a2cf95 in term 0.
I20260927 03:28:54.059724 11006 leader_election.cc:302] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66501d03b6f54ca6aa1016e82bd65358, c27391f44d0049c18be482bc48a2cf95; no voters:
I20260927 03:28:54.059710 10936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f134495c3647f9b1216fcb268adebe" candidate_uuid: "c27391f44d0049c18be482bc48a2cf95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24204c579ecb4926bb8e8d07ed4d68f2" is_pre_election: true
I20260927 03:28:54.059866 10936 raft_consensus.cc:2454] T e0f134495c3647f9b1216fcb268adebe P 24204c579ecb4926bb8e8d07ed4d68f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c27391f44d0049c18be482bc48a2cf95 in term 0.
I20260927 03:28:54.059983 11140 raft_consensus.cc:2781] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:28:54.060050 11140 raft_consensus.cc:482] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:28:54.060088 11140 raft_consensus.cc:3037] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:54.060812 11140 raft_consensus.cc:504] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24204c579ecb4926bb8e8d07ed4d68f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39673 } } peers { permanent_uuid: "66501d03b6f54ca6aa1016e82bd65358" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39795 } } peers { permanent_uuid: "c27391f44d0049c18be482bc48a2cf95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41085 } }
I20260927 03:28:54.060962 11140 leader_election.cc:288] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95 [CANDIDATE]: Term 1 election: Requested vote from peers 24204c579ecb4926bb8e8d07ed4d68f2 (127.0.98.2:39673), 66501d03b6f54ca6aa1016e82bd65358 (127.0.98.1:39795)
I20260927 03:28:54.061144 10802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f134495c3647f9b1216fcb268adebe" candidate_uuid: "c27391f44d0049c18be482bc48a2cf95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66501d03b6f54ca6aa1016e82bd65358"
I20260927 03:28:54.061170 10936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f134495c3647f9b1216fcb268adebe" candidate_uuid: "c27391f44d0049c18be482bc48a2cf95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24204c579ecb4926bb8e8d07ed4d68f2"
I20260927 03:28:54.061233 10936 raft_consensus.cc:3037] T e0f134495c3647f9b1216fcb268adebe P 24204c579ecb4926bb8e8d07ed4d68f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:54.061229 10802 raft_consensus.cc:3037] T e0f134495c3647f9b1216fcb268adebe P 66501d03b6f54ca6aa1016e82bd65358 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:28:54.062112 10802 raft_consensus.cc:2454] T e0f134495c3647f9b1216fcb268adebe P 66501d03b6f54ca6aa1016e82bd65358 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c27391f44d0049c18be482bc48a2cf95 in term 1.
I20260927 03:28:54.062112 10936 raft_consensus.cc:2454] T e0f134495c3647f9b1216fcb268adebe P 24204c579ecb4926bb8e8d07ed4d68f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c27391f44d0049c18be482bc48a2cf95 in term 1.
I20260927 03:28:54.062309 11004 leader_election.cc:302] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24204c579ecb4926bb8e8d07ed4d68f2, c27391f44d0049c18be482bc48a2cf95; no voters:
I20260927 03:28:54.062407 11140 raft_consensus.cc:2781] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:28:54.062567 11140 raft_consensus.cc:686] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95 [term 1 LEADER]: Becoming Leader. State: Replica: c27391f44d0049c18be482bc48a2cf95, State: Running, Role: LEADER
I20260927 03:28:54.062687 11140 consensus_queue.cc:237] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24204c579ecb4926bb8e8d07ed4d68f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39673 } } peers { permanent_uuid: "66501d03b6f54ca6aa1016e82bd65358" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39795 } } peers { permanent_uuid: "c27391f44d0049c18be482bc48a2cf95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41085 } }
I20260927 03:28:54.063653 10657 catalog_manager.cc:5495] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95 reported cstate change: term changed from 0 to 1, leader changed from <none> to c27391f44d0049c18be482bc48a2cf95 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c27391f44d0049c18be482bc48a2cf95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24204c579ecb4926bb8e8d07ed4d68f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66501d03b6f54ca6aa1016e82bd65358" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c27391f44d0049c18be482bc48a2cf95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41085 } health_report { overall_health: HEALTHY } } }
W20260927 03:28:54.070955 11117 tablet.cc:2369] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:28:54.083303 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:28:54.081071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60240 (local address 127.0.98.3:41085)
0927 03:28:54.081116 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:28:54.081118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:28:54.081145 (+ 27us) server_negotiation.cc:407] Connection header received
0927 03:28:54.081209 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:28:54.081210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:28:54.081228 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:28:54.081290 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:28:54.081388 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:54.081845 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:54.082633 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:28:54.082829 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:28:54.082885 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:28:54.082995 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:28:54.083088 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:28:54.083143 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:28:54.083214 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 03:28:54.083231 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:28:54.085033 10802 raft_consensus.cc:1261] T e0f134495c3647f9b1216fcb268adebe P 66501d03b6f54ca6aa1016e82bd65358 [term 1 FOLLOWER]: Refusing update from remote peer c27391f44d0049c18be482bc48a2cf95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:54.085047 10936 raft_consensus.cc:1261] T e0f134495c3647f9b1216fcb268adebe P 24204c579ecb4926bb8e8d07ed4d68f2 [term 1 FOLLOWER]: Refusing update from remote peer c27391f44d0049c18be482bc48a2cf95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:28:54.085371 11143 consensus_queue.cc:1035] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66501d03b6f54ca6aa1016e82bd65358" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:54.085516 11140 consensus_queue.cc:1035] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24204c579ecb4926bb8e8d07ed4d68f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:28:54.087767 11146 mvcc.cc:205] Tried to move back new op lower bound from 7333804990809559040 to 7333804990721490944. Current Snapshot: MvccSnapshot[applied={T|T < 7333804990809559040}]
I20260927 03:28:54.088581 11147 mvcc.cc:205] Tried to move back new op lower bound from 7333804990809559040 to 7333804990721490944. Current Snapshot: MvccSnapshot[applied={T|T < 7333804990809559040}]
I20260927 03:28:54.090152 10656 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:28:54.090255 10656 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:28:54.090983 10656 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef49c6899b81429fb0e64b151465b1d6: Sending DeleteTablet for 3 replicas of tablet e0f134495c3647f9b1216fcb268adebe
I20260927 03:28:54.091315 11050 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f134495c3647f9b1216fcb268adebe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60228
I20260927 03:28:54.091325 10782 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f134495c3647f9b1216fcb268adebe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42104
I20260927 03:28:54.091346 10916 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f134495c3647f9b1216fcb268adebe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:28:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41132
I20260927 03:28:54.091532 11155 tablet_replica.cc:330] T e0f134495c3647f9b1216fcb268adebe P 66501d03b6f54ca6aa1016e82bd65358: stopping tablet replica
I20260927 03:28:54.091639 11155 raft_consensus.cc:2229] T e0f134495c3647f9b1216fcb268adebe P 66501d03b6f54ca6aa1016e82bd65358 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:28:54.091699 11157 tablet_replica.cc:330] T e0f134495c3647f9b1216fcb268adebe P 24204c579ecb4926bb8e8d07ed4d68f2: stopping tablet replica
I20260927 03:28:54.091787 11155 raft_consensus.cc:2258] T e0f134495c3647f9b1216fcb268adebe P 66501d03b6f54ca6aa1016e82bd65358 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:54.091858 11157 raft_consensus.cc:2229] T e0f134495c3647f9b1216fcb268adebe P 24204c579ecb4926bb8e8d07ed4d68f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:28:54.091889 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 10719
I20260927 03:28:54.092742 11157 raft_consensus.cc:2258] T e0f134495c3647f9b1216fcb268adebe P 24204c579ecb4926bb8e8d07ed4d68f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:54.093211 11157 ts_tablet_manager.cc:1905] T e0f134495c3647f9b1216fcb268adebe P 24204c579ecb4926bb8e8d07ed4d68f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:28:54.094302 11156 tablet_replica.cc:330] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95: stopping tablet replica
I20260927 03:28:54.094444 11156 raft_consensus.cc:2229] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:28:54.094600 11156 raft_consensus.cc:2258] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:28:54.094904 11157 ts_tablet_manager.cc:1918] T e0f134495c3647f9b1216fcb268adebe P 24204c579ecb4926bb8e8d07ed4d68f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:28:54.094980 11157 log.cc:1196] T e0f134495c3647f9b1216fcb268adebe P 24204c579ecb4926bb8e8d07ed4d68f2: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-1/wal/wals/e0f134495c3647f9b1216fcb268adebe
I20260927 03:28:54.095311 11157 ts_tablet_manager.cc:1939] T e0f134495c3647f9b1216fcb268adebe P 24204c579ecb4926bb8e8d07ed4d68f2: Deleting consensus metadata
I20260927 03:28:54.095741 10642 catalog_manager.cc:4841] TS 24204c579ecb4926bb8e8d07ed4d68f2 (127.0.98.2:39673): tablet e0f134495c3647f9b1216fcb268adebe (table test-table [id=312d01e4c38f4d8ebbaac17429ceba85]) successfully deleted
I20260927 03:28:54.095800 11156 ts_tablet_manager.cc:1905] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 03:28:54.097482 10644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39795 (error 108)
I20260927 03:28:54.097664 11156 ts_tablet_manager.cc:1918] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:28:54.097821 11156 log.cc:1196] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790479675865103-392-0/minicluster-data/ts-2/wal/wals/e0f134495c3647f9b1216fcb268adebe
W20260927 03:28:54.098043 11127 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:28:54.097902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39795 (local address 127.0.0.1:42108)
0927 03:28:54.097942 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 03:28:54.097954 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:28:54.097994 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39795: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":20}
W20260927 03:28:54.098217 10644 catalog_manager.cc:4584] TS 66501d03b6f54ca6aa1016e82bd65358 (127.0.98.1:39795): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e0f134495c3647f9b1216fcb268adebe: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39795: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260927 03:28:54.098295 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 10853
I20260927 03:28:54.098941 11156 ts_tablet_manager.cc:1939] T e0f134495c3647f9b1216fcb268adebe P c27391f44d0049c18be482bc48a2cf95: Deleting consensus metadata
I20260927 03:28:54.099318 10643 catalog_manager.cc:4841] TS c27391f44d0049c18be482bc48a2cf95 (127.0.98.3:41085): tablet e0f134495c3647f9b1216fcb268adebe (table test-table [id=312d01e4c38f4d8ebbaac17429ceba85]) successfully deleted
I20260927 03:28:54.103803 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 10987
I20260927 03:28:54.109987 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 10625
2026-09-27T03:28:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2125 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:28:54 dist-test-slave-j0hf krb5kdc[11161](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:28:54 dist-test-slave-j0hf krb5kdc[11161](info): set up 2 sockets
Sep 27 03:28:54 dist-test-slave-j0hf krb5kdc[11161](info): commencing operation
krb5kdc: starting...
W20260927 03:28:55.176450 392 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:28:55 dist-test-slave-j0hf krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:28:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:28:55Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 03:28:55.337950 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task7YYkzb/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/postgres -l logfile start
2026-09-27 03:28:57.814 UTC [11191] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 03:28:57.814 UTC [11191] LOG: listening on IPv4 address "127.0.98.20", port 44875
2026-09-27 03:28:57.817 UTC [11191] LOG: listening on Unix socket "/tmp/.s.PGSQL.44875"
2026-09-27 03:28:57.839 UTC [11194] LOG: database system was shut down at 2026-09-27 03:28:56 UTC
2026-09-27 03:28:57.842 UTC [11191] LOG: database system is ready to accept connections
I20260927 03:28:58.804477 392 mini_postgres.cc:96] Postgres bound to 44875
2026-09-27 03:28:58.811 UTC [11202] FATAL: database "slave" does not exist
127.0.98.20:44875 - accepting connections
I20260927 03:28:58.811645 392 mini_ranger.cc:161] Starting Ranger...
I20260927 03:28:58.826287 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 03:28:59.334457 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 03:28:59.334524 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ranger-admin
2026-09-27 03:28:59,602 [I] DB FLAVOR :POSTGRES
2026-09-27 03:28:59,603 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 03:28:59,603 [I] Checking connection..
2026-09-27 03:28:59,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 03:28:59,862 [I] Checking connection passed.
2026-09-27 03:28:59,862 [I] --------- Verifying version history table ---------
2026-09-27 03:28:59,862 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 03:29:00,137 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 03:29:00,138 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 03:29:00,377 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 03:29:00,377 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 03:29:00,377 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44875/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 03:29:00.624 UTC [11266] WARNING: there is no transaction in progress
2026-09-27 03:29:00,630 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 03:29:00,630 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 03:29:00,857 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 03:29:00,857 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 03:29:00,858 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 03:29:01,105 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 03:29:01,353 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j0hf.c.gcp-upstream.internal','N') ;"
2026-09-27 03:29:01,570 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 03:29:01,797 [I] Table x_portal_user does not exist in database ranger
2026-09-27 03:29:01,798 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 03:29:02,046 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 03:29:02,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 03:29:02,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 03:29:02,544 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 03:29:02,544 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44875/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 03:29:03.496 UTC [11374] WARNING: there is no transaction in progress
2026-09-27 03:29:03.512 UTC [11374] WARNING: there is no transaction in progress
2026-09-27 03:29:03.529 UTC [11374] WARNING: there is no transaction in progress
2026-09-27 03:29:03.545 UTC [11374] WARNING: there is no transaction in progress
2026-09-27 03:29:03.564 UTC [11374] WARNING: there is no transaction in progress
2026-09-27 03:29:03.583 UTC [11374] WARNING: there is no transaction in progress
2026-09-27 03:29:03.640 UTC [11374] WARNING: there is no transaction in progress
2026-09-27 03:29:03.647 UTC [11374] WARNING: there is no transaction in progress
2026-09-27 03:29:03.662 UTC [11374] WARNING: there is no transaction in progress
2026-09-27 03:29:03.673 UTC [11374] WARNING: there is no transaction in progress
2026-09-27 03:29:03.683 UTC [11374] WARNING: there is no transaction in progress
2026-09-27 03:29:03.697 UTC [11374] WARNING: there is no transaction in progress
2026-09-27 03:29:03.707 UTC [11374] WARNING: there is no transaction in progress
2026-09-27 03:29:03.713 UTC [11374] WARNING: there is no transaction in progress
2026-09-27 03:29:03.720 UTC [11374] WARNING: there is no transaction in progress
2026-09-27 03:29:03.726 UTC [11374] WARNING: there is no transaction in progress
2026-09-27 03:29:04,384 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 03:29:04,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 03:29:04,613 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 03:29:04,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 03:29:04,841 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 03:29:04,841 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 03:29:05,094 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 03:29:05,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 03:29:05,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 03:29:05,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j0hf.c.gcp-upstream.internal';"
2026-09-27 03:29:05,826 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 03:29:05,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 03:29:06,087 [I] DB_PATCHES have already been applied
I20260927 03:29:06.093045 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task7YYkzb/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 03:29:06.093155 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 03:29:06.096280 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:47825
Sep 27, 2026 3:29:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:29:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:29:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 3:29:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 3:29:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 3:29:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 3:29:06 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 03:29:07.207640 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.396s user 0.000s sys 0.009s
Sep 27, 2026 3:29:27 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 3:29:27 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 3:29:27 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 3:29:27 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 3:29:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 3:29:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 3:29:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 3:29:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 3:29:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 3:29:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 3:29:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 3:29:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 3:29:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 3:29:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 3:29:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 3:29:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 3:29:27 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 3:29:28 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 03:29:28.645658 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 03:29:28 dist-test-slave-j0hf krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479768, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 03:29:28.727581 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 03:29:28.736821 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 03:29:29.589486 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 03:29:29.589576 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ranger-kms
I20260927 03:29:29.589599 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:44875
2026-09-27 03:29:29,767 [I] DB FLAVOR :POSTGRES
2026-09-27 03:29:29,767 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 03:29:29,767 [I] Checking connection
2026-09-27 03:29:29,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44875/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 03:29:30,017 [I] connection success
2026-09-27 03:29:30,017 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 03:29:30,017 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 03:29:30,017 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44875/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 03:29:30,245 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 03:29:30,246 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 03:29:30,246 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 03:29:30,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44875/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 03:29:30,497 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 03:29:30.949183 392 mini_ranger_kms.cc:310] Created kms service
I20260927 03:29:31.120533 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 03:29:31.163800 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 03:29:31 dist-test-slave-j0hf krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479768, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 03:29:31.572418 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 03:29:31.572582 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 03:29:31.572614 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 03:29:31.575856 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11554
I20260927 03:29:31.575999 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:34071
Sep 27, 2026 3:29:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:29:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:29:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 3:29:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 3:29:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 3:29:33 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 03:29:33.839537 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.112s user 0.003s sys 0.157s
I20260927 03:29:33.839650 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 03:29:33.839772 392 mini_ranger_kms.cc:401] 127.0.98.20:34071/kms/v1/keys
Sep 27 03:29:34 dist-test-slave-j0hf krb5kdc[11161](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790479774, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 03:29:34 dist-test-slave-j0hf krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479774, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 03:29:34 dist-test-slave-j0hf krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479774, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 03:29:37 dist-test-slave-j0hf krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:29:37.033818 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37775
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37775
--ranger_config_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34071/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:29:37.138573 11605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:29:37.138821 11605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:29:37.138845 11605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:29:37.140131 11605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:29:37.140178 11605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:29:37.140192 11605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:29:37.140205 11605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:29:37.140218 11605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:29:37.141871 11605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:34071/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37775
--ranger_config_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:29:37.142140 11605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:29:37.142390 11605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:29:37.147768 11611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:29:37.147878 11605 server_base.cc:1034] running on GCE node
W20260927 03:29:37.147742 11613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:29:37.147742 11610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:29:37.148245 11605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:29:37.148579 11605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:29:37.149737 11605 hybrid_clock.cc:648] HybridClock initialized: now 1790479777149715 us; error 37 us; skew 500 ppm
Sep 27 03:29:37 dist-test-slave-j0hf krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:29:37.157577 11605 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:29:37.158108 11605 webserver.cc:460] Webserver started at http://127.0.98.62:33869/ using document root <none> and password file <none>
I20260927 03:29:37.158352 11605 fs_manager.cc:359] Metadata directory not provided
I20260927 03:29:37.158414 11605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:29:37.158582 11605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:29:37 dist-test-slave-j0hf krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 03:29:37.204028 11605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "d7aab8afbce742298b1cd7680ddf40f5"
format_stamp: "Formatted at 2026-09-27 03:29:37 on dist-test-slave-j0hf"
server_key: "756b4cf7e6d5732475fd563b57ab46ab"
server_key_iv: "0cbcafaee970f353283fc2d626176f7b"
server_key_version: "kuduclusterkey@0"
I20260927 03:29:37.204468 11605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "d7aab8afbce742298b1cd7680ddf40f5"
format_stamp: "Formatted at 2026-09-27 03:29:37 on dist-test-slave-j0hf"
server_key: "756b4cf7e6d5732475fd563b57ab46ab"
server_key_iv: "0cbcafaee970f353283fc2d626176f7b"
server_key_version: "kuduclusterkey@0"
I20260927 03:29:37.205958 11605 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:29:37.220647 11620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:29:37.220927 11605 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:29:37.221050 11605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "d7aab8afbce742298b1cd7680ddf40f5"
format_stamp: "Formatted at 2026-09-27 03:29:37 on dist-test-slave-j0hf"
server_key: "756b4cf7e6d5732475fd563b57ab46ab"
server_key_iv: "0cbcafaee970f353283fc2d626176f7b"
server_key_version: "kuduclusterkey@0"
I20260927 03:29:37.221151 11605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:29:37.243875 11605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:29:37.246093 11605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:29:37.246263 11605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:29:37.251524 11605 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37775
I20260927 03:29:37.251578 11672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37775 every 8 connection(s)
I20260927 03:29:37.251957 11605 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:29:37.252635 11673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:29:37.254878 11673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7aab8afbce742298b1cd7680ddf40f5: Bootstrap starting.
I20260927 03:29:37.255538 11673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7aab8afbce742298b1cd7680ddf40f5: Neither blocks nor log segments found. Creating new log.
I20260927 03:29:37.255815 11673 log.cc:824] T 00000000000000000000000000000000 P d7aab8afbce742298b1cd7680ddf40f5: Log is configured to *not* fsync() on all Append() calls
I20260927 03:29:37.256493 11673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7aab8afbce742298b1cd7680ddf40f5: No bootstrap required, opened a new log
I20260927 03:29:37.258478 11673 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7aab8afbce742298b1cd7680ddf40f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7aab8afbce742298b1cd7680ddf40f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37775 } }
I20260927 03:29:37.258615 11673 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7aab8afbce742298b1cd7680ddf40f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:29:37.258637 11673 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7aab8afbce742298b1cd7680ddf40f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7aab8afbce742298b1cd7680ddf40f5, State: Initialized, Role: FOLLOWER
I20260927 03:29:37.258792 11673 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7aab8afbce742298b1cd7680ddf40f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7aab8afbce742298b1cd7680ddf40f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37775 } }
I20260927 03:29:37.258867 11673 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7aab8afbce742298b1cd7680ddf40f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:29:37.258890 11673 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7aab8afbce742298b1cd7680ddf40f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:29:37.258955 11673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7aab8afbce742298b1cd7680ddf40f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:29:37.259546 11673 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7aab8afbce742298b1cd7680ddf40f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7aab8afbce742298b1cd7680ddf40f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37775 } }
I20260927 03:29:37.259634 11673 leader_election.cc:302] T 00000000000000000000000000000000 P d7aab8afbce742298b1cd7680ddf40f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7aab8afbce742298b1cd7680ddf40f5; no voters:
I20260927 03:29:37.259836 11673 leader_election.cc:288] T 00000000000000000000000000000000 P d7aab8afbce742298b1cd7680ddf40f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:29:37.259856 11676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7aab8afbce742298b1cd7680ddf40f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:29:37.259966 11676 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7aab8afbce742298b1cd7680ddf40f5 [term 1 LEADER]: Becoming Leader. State: Replica: d7aab8afbce742298b1cd7680ddf40f5, State: Running, Role: LEADER
I20260927 03:29:37.260094 11673 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7aab8afbce742298b1cd7680ddf40f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:29:37.260365 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 11605
I20260927 03:29:37.260468 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:29:37.260677 392 external_mini_cluster.cc:1428] Setting key 5f4166ddccff590e5fd77c117d816c81
I20260927 03:29:37.260113 11676 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7aab8afbce742298b1cd7680ddf40f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7aab8afbce742298b1cd7680ddf40f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37775 } }
I20260927 03:29:37.261721 11677 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7aab8afbce742298b1cd7680ddf40f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7aab8afbce742298b1cd7680ddf40f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7aab8afbce742298b1cd7680ddf40f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37775 } } }
I20260927 03:29:37.261996 11677 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7aab8afbce742298b1cd7680ddf40f5 [sys.catalog]: This master's current role is: LEADER
I20260927 03:29:37.261935 11678 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7aab8afbce742298b1cd7680ddf40f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7aab8afbce742298b1cd7680ddf40f5. Latest consensus state: current_term: 1 leader_uuid: "d7aab8afbce742298b1cd7680ddf40f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7aab8afbce742298b1cd7680ddf40f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37775 } } }
I20260927 03:29:37.262261 11678 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7aab8afbce742298b1cd7680ddf40f5 [sys.catalog]: This master's current role is: LEADER
I20260927 03:29:37.262368 11673 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 03:29:37.262257 11693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:29:37.266454 11693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:29:37.268790 11693 catalog_manager.cc:1347] Generated new cluster ID: 51e7dc118e2f4530a8d3920414ad04b0
I20260927 03:29:37.268929 11693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:29:37 dist-test-slave-j0hf krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:29:37.276371 11683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:29:37.261335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47690 (local address 127.0.98.62:37775)
0927 03:29:37.262330 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0927 03:29:37.262340 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:29:37.262510 (+ 170us) server_negotiation.cc:407] Connection header received
0927 03:29:37.262678 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:29:37.262686 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:29:37.262749 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:29:37.262899 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:29:37.265586 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:29:37.267305 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:29:37.268438 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:29:37.269047 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:29:37.270943 (+ 1896us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:29:37.272261 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:29:37.272266 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:29:37.272267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:29:37.272280 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:29:37.274327 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:29:37.274723 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:29:37.274726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:29:37.274726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:29:37.274823 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:29:37.275014 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:29:37.275014 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:29:37.275015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:29:37.275332 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:29:37.275936 (+ 604us) server_negotiation.cc:1035] Waiting for connection context
0927 03:29:37.276009 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 03:29:37.276145 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":942,"thread_start_us":67,"threads_started":1}
I20260927 03:29:37.279615 11693 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:29:37.280350 11693 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:29:37.288143 11693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7aab8afbce742298b1cd7680ddf40f5: Generated new TSK 0
I20260927 03:29:37.288422 11693 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:29:38.039211 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34071/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37775
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:29:38.149039 11718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:29:38.149232 11718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:29:38.149256 11718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:29:38.150481 11718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:29:38.150522 11718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:29:38.150549 11718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:29:38.152648 11718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:34071/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:29:38.153046 11718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:29:38.153354 11718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:29:38.158601 11724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:29:38.158766 11718 server_base.cc:1034] running on GCE node
W20260927 03:29:38.158741 11726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:29:38.158596 11723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:29:38.159142 11718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:29:38.159452 11718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:29:38.160641 11718 hybrid_clock.cc:648] HybridClock initialized: now 1790479778160633 us; error 35 us; skew 500 ppm
Sep 27 03:29:38 dist-test-slave-j0hf krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:29:38.163757 11718 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:29:38.164259 11718 webserver.cc:460] Webserver started at http://127.0.98.1:35995/ using document root <none> and password file <none>
I20260927 03:29:38.164495 11718 fs_manager.cc:359] Metadata directory not provided
I20260927 03:29:38.164565 11718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:29:38.164729 11718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:29:38 dist-test-slave-j0hf krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 03:29:38.178726 11718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-0/data/instance:
uuid: "2241c6bc25fb4d4c9a3141fe28f2cc37"
format_stamp: "Formatted at 2026-09-27 03:29:38 on dist-test-slave-j0hf"
server_key: "09acdf39cd661d9919135de7bebe7261"
server_key_iv: "88a1d0b6195864254bad0c42b1674db2"
server_key_version: "kuduclusterkey@0"
I20260927 03:29:38.179203 11718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2241c6bc25fb4d4c9a3141fe28f2cc37"
format_stamp: "Formatted at 2026-09-27 03:29:38 on dist-test-slave-j0hf"
server_key: "09acdf39cd661d9919135de7bebe7261"
server_key_iv: "88a1d0b6195864254bad0c42b1674db2"
server_key_version: "kuduclusterkey@0"
I20260927 03:29:38.181023 11718 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 03:29:38.194082 11735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:29:38.194314 11718 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:29:38.194397 11718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "2241c6bc25fb4d4c9a3141fe28f2cc37"
format_stamp: "Formatted at 2026-09-27 03:29:38 on dist-test-slave-j0hf"
server_key: "09acdf39cd661d9919135de7bebe7261"
server_key_iv: "88a1d0b6195864254bad0c42b1674db2"
server_key_version: "kuduclusterkey@0"
I20260927 03:29:38.194482 11718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:29:38.228618 11718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:29:38.231155 11718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:29:38.231375 11718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:29:38.231649 11718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:29:38.232053 11718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:29:38.232138 11718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:29:38.232308 11718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:29:38.232381 11718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:29:38.239466 11718 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42559
I20260927 03:29:38.240001 11718 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:29:38.240224 11848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42559 every 8 connection(s)
I20260927 03:29:38.247237 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 11718
I20260927 03:29:38.247361 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:29:38.247596 392 external_mini_cluster.cc:1428] Setting key 2386f513e74c37b3333977cd9494584b
Sep 27 03:29:38 dist-test-slave-j0hf krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:29:38.253381 11851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:29:38.240919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37775 (local address 127.0.98.1:59405)
0927 03:29:38.241158 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 03:29:38.241207 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 03:29:38.241827 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:29:38.243824 (+ 1997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:29:38.243829 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:29:38.244080 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:29:38.244370 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:29:38.244376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:29:38.245398 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:29:38.245399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:29:38.246683 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:29:38.246686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:29:38.247272 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:29:38.247273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:29:38.247428 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:29:38.248140 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:29:38.248150 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:29:38.249732 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:29:38.251766 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:29:38.251772 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:29:38.251775 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:29:38.252062 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:29:38.252274 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:29:38.252275 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:29:38.252275 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:29:38.252394 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:29:38.252679 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:29:38.252684 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:29:38.252852 (+ 168us) client_negotiation.cc:769] Sending connection context
0927 03:29:38.252959 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 03:29:38.253120 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":90,"threads_started":1}
I20260927 03:29:38.253911 11852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:29:38.240970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59405 (local address 127.0.98.62:37775)
0927 03:29:38.243579 (+ 2609us) server_negotiation.cc:206] Beginning negotiation
0927 03:29:38.243581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:29:38.243591 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:29:38.243620 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:29:38.243621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:29:38.243634 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:29:38.243710 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:29:38.244488 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:29:38.245289 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:29:38.246783 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:29:38.247143 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:29:38.247206 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:29:38.249833 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:29:38.249835 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:29:38.249835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:29:38.249842 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:29:38.251681 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:29:38.252146 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:29:38.252147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:29:38.252147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:29:38.252211 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:29:38.252462 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:29:38.252463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:29:38.252463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:29:38.252593 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:29:38.252654 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:29:38.253735 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0927 03:29:38.253819 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2564,"thread_start_us":92,"threads_started":1}
I20260927 03:29:38.254604 11849 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37775
I20260927 03:29:38.254694 11849 heartbeater.cc:464] Registering TS with master...
I20260927 03:29:38.254889 11849 heartbeater.cc:511] Master 127.0.98.62:37775 requested a full tablet report, sending...
I20260927 03:29:38.255450 11637 ts_manager.cc:194] Registered new tserver with Master: 2241c6bc25fb4d4c9a3141fe28f2cc37 (127.0.98.1:42559)
I20260927 03:29:38.256338 11637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59405
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:29:38.314122 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34071/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37775
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 03:29:38 dist-test-slave-j0hf krb5kdc[11161](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790479778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 03:29:38.447559 11857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:29:38.447842 11857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:29:38.447877 11857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:29:38.449657 11857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:29:38.449715 11857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:29:38.449756 11857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:29:38.452006 11857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:34071/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:29:38.452524 11857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:29:38.452804 11857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:29:38.458926 11862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:29:38.459132 11863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:29:38.459022 11865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:29:38.459453 11857 server_base.cc:1034] running on GCE node
I20260927 03:29:38.459640 11857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:29:38.459894 11857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:29:38.461001 11857 hybrid_clock.cc:648] HybridClock initialized: now 1790479778460979 us; error 36 us; skew 500 ppm
Sep 27 03:29:38 dist-test-slave-j0hf krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:29:38.464270 11857 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:29:38.464811 11857 webserver.cc:460] Webserver started at http://127.0.98.2:38277/ using document root <none> and password file <none>
I20260927 03:29:38.465122 11857 fs_manager.cc:359] Metadata directory not provided
I20260927 03:29:38.465193 11857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:29:38.465364 11857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:29:38 dist-test-slave-j0hf krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 03:29:38.478847 11857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-1/data/instance:
uuid: "3081fca0a988479aa888dd8b771c04ea"
format_stamp: "Formatted at 2026-09-27 03:29:38 on dist-test-slave-j0hf"
server_key: "d81a7a809a3101fce25375639feb8bd7"
server_key_iv: "6cc49c76329c4c4e1bfcf78342e5fc82"
server_key_version: "kuduclusterkey@0"
I20260927 03:29:38.479313 11857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3081fca0a988479aa888dd8b771c04ea"
format_stamp: "Formatted at 2026-09-27 03:29:38 on dist-test-slave-j0hf"
server_key: "d81a7a809a3101fce25375639feb8bd7"
server_key_iv: "6cc49c76329c4c4e1bfcf78342e5fc82"
server_key_version: "kuduclusterkey@0"
I20260927 03:29:38.482743 11857 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260927 03:29:38.508073 11873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:29:38.508363 11857 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 03:29:38.508505 11857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "3081fca0a988479aa888dd8b771c04ea"
format_stamp: "Formatted at 2026-09-27 03:29:38 on dist-test-slave-j0hf"
server_key: "d81a7a809a3101fce25375639feb8bd7"
server_key_iv: "6cc49c76329c4c4e1bfcf78342e5fc82"
server_key_version: "kuduclusterkey@0"
I20260927 03:29:38.508749 11857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:29:38.537176 11857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:29:38.539755 11857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:29:38.539948 11857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:29:38.540223 11857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:29:38.540644 11857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:29:38.540740 11857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:29:38.540819 11857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:29:38.540851 11857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:29:38.547017 11857 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42051
I20260927 03:29:38.547550 11857 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:29:38.549495 11986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42051 every 8 connection(s)
I20260927 03:29:38.550926 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 11857
I20260927 03:29:38.551031 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:29:38.551222 392 external_mini_cluster.cc:1428] Setting key f23050aab01b2bd6c8795f49b5c1a1fd
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 27 03:29:38 dist-test-slave-j0hf krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:29:38.581475 11852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:29:38.552043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37849 (local address 127.0.98.62:37775)
0927 03:29:38.552089 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:29:38.552092 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:29:38.559735 (+ 7643us) server_negotiation.cc:407] Connection header received
0927 03:29:38.560514 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:29:38.560516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:29:38.560532 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:29:38.560591 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:29:38.561929 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:29:38.562700 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:29:38.565451 (+ 2751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:29:38.565770 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:29:38.565821 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:29:38.577670 (+ 11849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:29:38.577673 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:29:38.577673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:29:38.577683 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:29:38.579668 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:29:38.580255 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:29:38.580256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:29:38.580257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:29:38.580329 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:29:38.580632 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:29:38.580634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:29:38.580634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:29:38.580788 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:29:38.580847 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 03:29:38.581269 (+ 422us) server_negotiation.cc:299] Negotiation successful
0927 03:29:38.581370 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:29:38.582278 11989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:29:38.552063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37775 (local address 127.0.98.2:37849)
0927 03:29:38.559542 (+ 7479us) negotiation.cc:107] Waiting for socket to connect
0927 03:29:38.559590 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 03:29:38.560356 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:29:38.560896 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:29:38.560902 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:29:38.561160 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:29:38.561450 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:29:38.561457 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:29:38.563928 (+ 2471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:29:38.563929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:29:38.565342 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:29:38.565345 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:29:38.565922 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:29:38.565923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:29:38.566088 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:29:38.575606 (+ 9518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:29:38.575620 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:29:38.577488 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:29:38.579786 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:29:38.579796 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:29:38.579799 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:29:38.580158 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:29:38.580401 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:29:38.580401 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:29:38.580402 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:29:38.580551 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:29:38.580882 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:29:38.580890 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:29:38.581096 (+ 206us) client_negotiation.cc:769] Sending connection context
0927 03:29:38.581842 (+ 746us) client_negotiation.cc:240] Negotiation successful
0927 03:29:38.582035 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7410,"thread_start_us":83,"threads_started":1}
I20260927 03:29:38.583000 11987 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37775
I20260927 03:29:38.583096 11987 heartbeater.cc:464] Registering TS with master...
I20260927 03:29:38.583284 11987 heartbeater.cc:511] Master 127.0.98.62:37775 requested a full tablet report, sending...
I20260927 03:29:38.583793 11637 ts_manager.cc:194] Registered new tserver with Master: 3081fca0a988479aa888dd8b771c04ea (127.0.98.2:42051)
I20260927 03:29:38.584334 11637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37849
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:29:38.594135 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34071/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37775
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:29:38.711856 11992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:29:38.712205 11992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:29:38.712297 11992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:29:38.713788 11992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:29:38.713852 11992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:29:38.713898 11992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:29:38.716022 11992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:34071/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:29:38.716478 11992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:29:38.716773 11992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:29:38.723119 12000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:29:38.723129 11998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:29:38.723297 11997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:29:38.724635 11992 server_base.cc:1034] running on GCE node
I20260927 03:29:38.725250 11992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:29:38.725539 11992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:29:38.726733 11992 hybrid_clock.cc:648] HybridClock initialized: now 1790479778726713 us; error 38 us; skew 500 ppm
Sep 27 03:29:38 dist-test-slave-j0hf krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:29:38.729984 11992 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:29:38.730490 11992 webserver.cc:460] Webserver started at http://127.0.98.3:36165/ using document root <none> and password file <none>
I20260927 03:29:38.730815 11992 fs_manager.cc:359] Metadata directory not provided
I20260927 03:29:38.730878 11992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:29:38.731048 11992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:29:38 dist-test-slave-j0hf krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 03:29:38.747761 11992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-2/data/instance:
uuid: "15e0ef5da73245bba93caa21b1a15e2d"
format_stamp: "Formatted at 2026-09-27 03:29:38 on dist-test-slave-j0hf"
server_key: "dc5efadc8ed775275fa91e1a8970625c"
server_key_iv: "a97125604bcae5b25465a04aaee6048d"
server_key_version: "kuduclusterkey@0"
I20260927 03:29:38.748466 11992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-2/wal/instance:
uuid: "15e0ef5da73245bba93caa21b1a15e2d"
format_stamp: "Formatted at 2026-09-27 03:29:38 on dist-test-slave-j0hf"
server_key: "dc5efadc8ed775275fa91e1a8970625c"
server_key_iv: "a97125604bcae5b25465a04aaee6048d"
server_key_version: "kuduclusterkey@0"
I20260927 03:29:38.754989 11992 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.000s
I20260927 03:29:38.772423 12007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:29:38.772691 11992 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260927 03:29:38.772782 11992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "15e0ef5da73245bba93caa21b1a15e2d"
format_stamp: "Formatted at 2026-09-27 03:29:38 on dist-test-slave-j0hf"
server_key: "dc5efadc8ed775275fa91e1a8970625c"
server_key_iv: "a97125604bcae5b25465a04aaee6048d"
server_key_version: "kuduclusterkey@0"
I20260927 03:29:38.772878 11992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:29:38.794423 11992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:29:38.797304 11992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:29:38.797513 11992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:29:38.797770 11992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:29:38.798182 11992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:29:38.798271 11992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:29:38.798342 11992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:29:38.798398 11992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:29:38.805212 11992 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42423
I20260927 03:29:38.805742 11992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:29:38.813858 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 11992
I20260927 03:29:38.813961 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:29:38.814165 392 external_mini_cluster.cc:1428] Setting key f674d0f6a4fd5f0d75833430a35a4876
I20260927 03:29:38.817320 12120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42423 every 8 connection(s)
Sep 27 03:29:38 dist-test-slave-j0hf krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:29:38.832257 11852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:29:38.818983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43007 (local address 127.0.98.62:37775)
0927 03:29:38.819058 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 03:29:38.819061 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:29:38.819697 (+ 636us) server_negotiation.cc:407] Connection header received
0927 03:29:38.820427 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:29:38.820429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:29:38.820442 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:29:38.820475 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:29:38.821364 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:29:38.822156 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:29:38.823735 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:29:38.824041 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:29:38.824091 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:29:38.828607 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:29:38.828609 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:29:38.828609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:29:38.828618 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:29:38.830499 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:29:38.831016 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:29:38.831018 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:29:38.831018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:29:38.831092 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:29:38.831398 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:29:38.831399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:29:38.831399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:29:38.831580 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:29:38.831639 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 03:29:38.832075 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 03:29:38.832160 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 03:29:38.832850 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:29:38.818671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37775 (local address 127.0.98.3:43007)
0927 03:29:38.819557 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0927 03:29:38.819607 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 03:29:38.820297 (+ 690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:29:38.820674 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:29:38.820679 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:29:38.820947 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:29:38.821246 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:29:38.821253 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:29:38.822285 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:29:38.822286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:29:38.823631 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:29:38.823634 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:29:38.824214 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:29:38.824215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:29:38.824368 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:29:38.826725 (+ 2357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:29:38.826736 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:29:38.828450 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:29:38.830597 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:29:38.830605 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:29:38.830608 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:29:38.830924 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:29:38.831167 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:29:38.831167 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:29:38.831168 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:29:38.831312 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:29:38.831675 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:29:38.831681 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:29:38.831870 (+ 189us) client_negotiation.cc:769] Sending connection context
0927 03:29:38.832369 (+ 499us) client_negotiation.cc:240] Negotiation successful
0927 03:29:38.832550 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"thread_start_us":109,"threads_started":1}
I20260927 03:29:38.834338 12121 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37775
I20260927 03:29:38.834458 12121 heartbeater.cc:464] Registering TS with master...
I20260927 03:29:38.834656 12121 heartbeater.cc:511] Master 127.0.98.62:37775 requested a full tablet report, sending...
I20260927 03:29:38.835057 11637 ts_manager.cc:194] Registered new tserver with Master: 15e0ef5da73245bba93caa21b1a15e2d (127.0.98.3:42423)
I20260927 03:29:38.835613 11637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43007
I20260927 03:29:38.837257 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:29:38.847484 11852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:29:38.838942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47692 (local address 127.0.98.62:37775)
0927 03:29:38.839048 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 03:29:38.839050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:29:38.839405 (+ 355us) server_negotiation.cc:407] Connection header received
0927 03:29:38.839549 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:29:38.839549 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:29:38.839560 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:29:38.839588 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:29:38.840346 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:29:38.841124 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:29:38.842264 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:29:38.842552 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:29:38.842600 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:29:38.844314 (+ 1714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:29:38.844315 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:29:38.844316 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:29:38.844323 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:29:38.846127 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:29:38.846549 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:29:38.846550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:29:38.846550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:29:38.846610 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:29:38.846817 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:29:38.846818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:29:38.846818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:29:38.846959 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:29:38.847043 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 03:29:38.847259 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 03:29:38.847340 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 03:29:38.849951 11637 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:29:38.852253 11637 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:29:38.868319 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:29:38.856250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42051 (local address 127.0.0.1:32900)
0927 03:29:38.856988 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0927 03:29:38.856995 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 03:29:38.857036 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:29:38.864350 (+ 7314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:29:38.864351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:29:38.864353 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:29:38.864404 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:29:38.864406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:29:38.865685 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:29:38.865686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:29:38.866842 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:29:38.866845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:29:38.867968 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:29:38.867969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:29:38.868062 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:29:38.868077 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 03:29:38.868151 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 03:29:38.868213 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":56,"threads_started":1}
I20260927 03:29:38.868383 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:29:38.856216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44802 (local address 127.0.98.1:42559)
0927 03:29:38.863533 (+ 7317us) server_negotiation.cc:206] Beginning negotiation
0927 03:29:38.863539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:29:38.863553 (+ 14us) server_negotiation.cc:407] Connection header received
0927 03:29:38.863596 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:29:38.863599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:29:38.863629 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:29:38.863734 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:29:38.864044 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:29:38.865214 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:29:38.867037 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:29:38.867864 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:29:38.867933 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:29:38.867988 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:29:38.868267 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 03:29:38.868309 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7272,"thread_start_us":58,"threads_started":1}
I20260927 03:29:38.868857 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:29:38.856110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42559 (local address 127.0.0.1:44802)
0927 03:29:38.856889 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0927 03:29:38.856896 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 03:29:38.856927 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:29:38.863848 (+ 6921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:29:38.863852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:29:38.863855 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:29:38.863936 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:29:38.863938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:29:38.865314 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:29:38.865314 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:29:38.866659 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:29:38.866662 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:29:38.868034 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:29:38.868035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:29:38.868103 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:29:38.868107 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:29:38.868155 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 03:29:38.868214 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"mutex_wait_us":16,"thread_start_us":62,"threads_started":1}
I20260927 03:29:38.869726 11783 tablet_service.cc:1467] Processing CreateTablet for tablet 249df48005ed40a6acad43f9e97fd6b7 (DEFAULT_TABLE table=test-table [id=5c15235e72eb481891af106f66e4bcc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:29:38.870097 11783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 249df48005ed40a6acad43f9e97fd6b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:29:38.871630 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:29:38.856442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32900 (local address 127.0.98.2:42051)
0927 03:29:38.864103 (+ 7661us) server_negotiation.cc:206] Beginning negotiation
0927 03:29:38.864109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:29:38.864120 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:29:38.864159 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:29:38.864161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:29:38.864177 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:29:38.864256 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:29:38.864484 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:29:38.865572 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:29:38.866962 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:29:38.867777 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:29:38.867850 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:29:38.867931 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 03:29:38.871491 (+ 3560us) server_negotiation.cc:299] Negotiation successful
0927 03:29:38.871541 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7617,"thread_start_us":71,"threads_started":1}
I20260927 03:29:38.872241 11915 tablet_service.cc:1467] Processing CreateTablet for tablet 249df48005ed40a6acad43f9e97fd6b7 (DEFAULT_TABLE table=test-table [id=5c15235e72eb481891af106f66e4bcc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:29:38.872555 11915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 249df48005ed40a6acad43f9e97fd6b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:29:38.874369 12140 tablet_bootstrap.cc:492] T 249df48005ed40a6acad43f9e97fd6b7 P 2241c6bc25fb4d4c9a3141fe28f2cc37: Bootstrap starting.
I20260927 03:29:38.875161 12140 tablet_bootstrap.cc:654] T 249df48005ed40a6acad43f9e97fd6b7 P 2241c6bc25fb4d4c9a3141fe28f2cc37: Neither blocks nor log segments found. Creating new log.
I20260927 03:29:38.875589 12140 log.cc:824] T 249df48005ed40a6acad43f9e97fd6b7 P 2241c6bc25fb4d4c9a3141fe28f2cc37: Log is configured to *not* fsync() on all Append() calls
I20260927 03:29:38.876070 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:29:38.855860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42423 (local address 127.0.0.1:43970)
0927 03:29:38.856691 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0927 03:29:38.856733 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:29:38.856796 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:29:38.871749 (+ 14953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:29:38.871750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:29:38.871753 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:29:38.871804 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:29:38.871806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:29:38.873640 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:29:38.873641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:29:38.874777 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:29:38.874780 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:29:38.875835 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:29:38.875836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:29:38.875905 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:29:38.875911 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 03:29:38.875942 (+ 31us) client_negotiation.cc:240] Negotiation successful
0927 03:29:38.875987 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":74,"threads_started":1}
I20260927 03:29:38.875978 12142 tablet_bootstrap.cc:492] T 249df48005ed40a6acad43f9e97fd6b7 P 3081fca0a988479aa888dd8b771c04ea: Bootstrap starting.
I20260927 03:29:38.876715 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:29:38.855952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43970 (local address 127.0.98.3:42423)
0927 03:29:38.871547 (+ 15595us) server_negotiation.cc:206] Beginning negotiation
0927 03:29:38.871552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:29:38.871566 (+ 14us) server_negotiation.cc:407] Connection header received
0927 03:29:38.871609 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:29:38.871611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:29:38.871627 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:29:38.871885 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:29:38.871966 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:29:38.873471 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:29:38.874917 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:29:38.875726 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:29:38.876449 (+ 723us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:29:38.876556 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 03:29:38.876585 (+ 29us) server_negotiation.cc:299] Negotiation successful
0927 03:29:38.876628 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15539,"thread_start_us":66,"threads_started":1}
I20260927 03:29:38.876974 12142 tablet_bootstrap.cc:654] T 249df48005ed40a6acad43f9e97fd6b7 P 3081fca0a988479aa888dd8b771c04ea: Neither blocks nor log segments found. Creating new log.
I20260927 03:29:38.877334 12142 log.cc:824] T 249df48005ed40a6acad43f9e97fd6b7 P 3081fca0a988479aa888dd8b771c04ea: Log is configured to *not* fsync() on all Append() calls
I20260927 03:29:38.877511 12055 tablet_service.cc:1467] Processing CreateTablet for tablet 249df48005ed40a6acad43f9e97fd6b7 (DEFAULT_TABLE table=test-table [id=5c15235e72eb481891af106f66e4bcc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:29:38.877866 12055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 249df48005ed40a6acad43f9e97fd6b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:29:38.878125 12140 tablet_bootstrap.cc:492] T 249df48005ed40a6acad43f9e97fd6b7 P 2241c6bc25fb4d4c9a3141fe28f2cc37: No bootstrap required, opened a new log
I20260927 03:29:38.878186 12140 ts_tablet_manager.cc:1397] T 249df48005ed40a6acad43f9e97fd6b7 P 2241c6bc25fb4d4c9a3141fe28f2cc37: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260927 03:29:38.880730 12140 raft_consensus.cc:348] T 249df48005ed40a6acad43f9e97fd6b7 P 2241c6bc25fb4d4c9a3141fe28f2cc37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2241c6bc25fb4d4c9a3141fe28f2cc37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42559 } } peers { permanent_uuid: "15e0ef5da73245bba93caa21b1a15e2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42423 } } peers { permanent_uuid: "3081fca0a988479aa888dd8b771c04ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42051 } }
I20260927 03:29:38.880894 12140 raft_consensus.cc:374] T 249df48005ed40a6acad43f9e97fd6b7 P 2241c6bc25fb4d4c9a3141fe28f2cc37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:29:38.880927 12140 raft_consensus.cc:729] T 249df48005ed40a6acad43f9e97fd6b7 P 2241c6bc25fb4d4c9a3141fe28f2cc37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2241c6bc25fb4d4c9a3141fe28f2cc37, State: Initialized, Role: FOLLOWER
I20260927 03:29:38.881451 12140 consensus_queue.cc:260] T 249df48005ed40a6acad43f9e97fd6b7 P 2241c6bc25fb4d4c9a3141fe28f2cc37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2241c6bc25fb4d4c9a3141fe28f2cc37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42559 } } peers { permanent_uuid: "15e0ef5da73245bba93caa21b1a15e2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42423 } } peers { permanent_uuid: "3081fca0a988479aa888dd8b771c04ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42051 } }
I20260927 03:29:38.881847 12140 ts_tablet_manager.cc:1428] T 249df48005ed40a6acad43f9e97fd6b7 P 2241c6bc25fb4d4c9a3141fe28f2cc37: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260927 03:29:38.882009 11849 heartbeater.cc:503] Master 127.0.98.62:37775 was elected leader, sending a full tablet report...
I20260927 03:29:38.883550 12142 tablet_bootstrap.cc:492] T 249df48005ed40a6acad43f9e97fd6b7 P 3081fca0a988479aa888dd8b771c04ea: No bootstrap required, opened a new log
I20260927 03:29:38.883677 12142 ts_tablet_manager.cc:1397] T 249df48005ed40a6acad43f9e97fd6b7 P 3081fca0a988479aa888dd8b771c04ea: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260927 03:29:38.885006 12144 tablet_bootstrap.cc:492] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d: Bootstrap starting.
I20260927 03:29:38.886019 12144 tablet_bootstrap.cc:654] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d: Neither blocks nor log segments found. Creating new log.
I20260927 03:29:38.886377 12144 log.cc:824] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d: Log is configured to *not* fsync() on all Append() calls
I20260927 03:29:38.886440 12142 raft_consensus.cc:348] T 249df48005ed40a6acad43f9e97fd6b7 P 3081fca0a988479aa888dd8b771c04ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2241c6bc25fb4d4c9a3141fe28f2cc37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42559 } } peers { permanent_uuid: "15e0ef5da73245bba93caa21b1a15e2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42423 } } peers { permanent_uuid: "3081fca0a988479aa888dd8b771c04ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42051 } }
I20260927 03:29:38.886691 12142 raft_consensus.cc:374] T 249df48005ed40a6acad43f9e97fd6b7 P 3081fca0a988479aa888dd8b771c04ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:29:38.886780 12142 raft_consensus.cc:729] T 249df48005ed40a6acad43f9e97fd6b7 P 3081fca0a988479aa888dd8b771c04ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3081fca0a988479aa888dd8b771c04ea, State: Initialized, Role: FOLLOWER
I20260927 03:29:38.886986 12142 consensus_queue.cc:260] T 249df48005ed40a6acad43f9e97fd6b7 P 3081fca0a988479aa888dd8b771c04ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2241c6bc25fb4d4c9a3141fe28f2cc37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42559 } } peers { permanent_uuid: "15e0ef5da73245bba93caa21b1a15e2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42423 } } peers { permanent_uuid: "3081fca0a988479aa888dd8b771c04ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42051 } }
I20260927 03:29:38.887179 12144 tablet_bootstrap.cc:492] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d: No bootstrap required, opened a new log
I20260927 03:29:38.887251 12144 ts_tablet_manager.cc:1397] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:29:38.887403 12142 ts_tablet_manager.cc:1428] T 249df48005ed40a6acad43f9e97fd6b7 P 3081fca0a988479aa888dd8b771c04ea: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 03:29:38.887898 11987 heartbeater.cc:503] Master 127.0.98.62:37775 was elected leader, sending a full tablet report...
I20260927 03:29:38.889809 12144 raft_consensus.cc:348] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2241c6bc25fb4d4c9a3141fe28f2cc37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42559 } } peers { permanent_uuid: "15e0ef5da73245bba93caa21b1a15e2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42423 } } peers { permanent_uuid: "3081fca0a988479aa888dd8b771c04ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42051 } }
I20260927 03:29:38.890007 12144 raft_consensus.cc:374] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:29:38.890057 12144 raft_consensus.cc:729] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15e0ef5da73245bba93caa21b1a15e2d, State: Initialized, Role: FOLLOWER
I20260927 03:29:38.890206 12144 consensus_queue.cc:260] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2241c6bc25fb4d4c9a3141fe28f2cc37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42559 } } peers { permanent_uuid: "15e0ef5da73245bba93caa21b1a15e2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42423 } } peers { permanent_uuid: "3081fca0a988479aa888dd8b771c04ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42051 } }
I20260927 03:29:38.890487 12144 ts_tablet_manager.cc:1428] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:29:38.890623 12121 heartbeater.cc:503] Master 127.0.98.62:37775 was elected leader, sending a full tablet report...
Sep 27 03:29:38 dist-test-slave-j0hf krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 03:29:38.992328 11850 tablet.cc:2369] T 249df48005ed40a6acad43f9e97fd6b7 P 2241c6bc25fb4d4c9a3141fe28f2cc37: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:29:39.052220 11988 tablet.cc:2369] T 249df48005ed40a6acad43f9e97fd6b7 P 3081fca0a988479aa888dd8b771c04ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:29:39.067384 12148 raft_consensus.cc:482] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:29:39.067555 12148 raft_consensus.cc:504] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2241c6bc25fb4d4c9a3141fe28f2cc37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42559 } } peers { permanent_uuid: "15e0ef5da73245bba93caa21b1a15e2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42423 } } peers { permanent_uuid: "3081fca0a988479aa888dd8b771c04ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42051 } }
I20260927 03:29:39.067992 12148 leader_election.cc:288] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2241c6bc25fb4d4c9a3141fe28f2cc37 (127.0.98.1:42559), 3081fca0a988479aa888dd8b771c04ea (127.0.98.2:42051)
W20260927 03:29:39.068238 12122 tablet.cc:2369] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:29:39.072286 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:29:39.068633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57513 (local address 127.0.98.2:42051)
0927 03:29:39.068685 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 03:29:39.068687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:29:39.068910 (+ 223us) server_negotiation.cc:407] Connection header received
0927 03:29:39.069000 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:29:39.069001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:29:39.069013 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:29:39.069039 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:29:39.069240 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:29:39.069968 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:29:39.071341 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:29:39.071924 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:29:39.071968 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:29:39.071979 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:29:39.072179 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 03:29:39.072214 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 03:29:39.072491 12149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:29:39.068269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42051 (local address 127.0.98.3:57513)
0927 03:29:39.068857 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0927 03:29:39.068866 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:29:39.068932 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:29:39.069095 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:29:39.069095 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:29:39.069097 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:29:39.069170 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:29:39.069171 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:29:39.070073 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:29:39.070073 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:29:39.071241 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:29:39.071243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:29:39.072033 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:29:39.072034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:29:39.072103 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:29:39.072106 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:29:39.072363 (+ 257us) client_negotiation.cc:240] Negotiation successful
0927 03:29:39.072403 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":72,"threads_started":1}
I20260927 03:29:39.072897 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:29:39.068163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42559 (local address 127.0.98.3:44117)
0927 03:29:39.068428 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 03:29:39.068439 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:29:39.068484 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:29:39.068816 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:29:39.068817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:29:39.068819 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:29:39.068891 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:29:39.068894 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:29:39.069854 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:29:39.069855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:29:39.071090 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:29:39.071092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:29:39.072321 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:29:39.072322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:29:39.072392 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:29:39.072396 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:29:39.072446 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 03:29:39.072489 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246}
I20260927 03:29:39.073441 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:29:39.068639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44117 (local address 127.0.98.1:42559)
0927 03:29:39.068681 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:29:39.068683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:29:39.068691 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:29:39.068719 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:29:39.068720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:29:39.068729 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:29:39.068784 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:29:39.069009 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:29:39.069753 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:29:39.071206 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:29:39.072203 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:29:39.072278 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:29:39.072288 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:29:39.073316 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0927 03:29:39.073365 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:29:39.073438 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "249df48005ed40a6acad43f9e97fd6b7" candidate_uuid: "15e0ef5da73245bba93caa21b1a15e2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3081fca0a988479aa888dd8b771c04ea" is_pre_election: true
I20260927 03:29:39.073604 11941 raft_consensus.cc:2454] T 249df48005ed40a6acad43f9e97fd6b7 P 3081fca0a988479aa888dd8b771c04ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15e0ef5da73245bba93caa21b1a15e2d in term 0.
I20260927 03:29:39.073685 11797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "249df48005ed40a6acad43f9e97fd6b7" candidate_uuid: "15e0ef5da73245bba93caa21b1a15e2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2241c6bc25fb4d4c9a3141fe28f2cc37" is_pre_election: true
I20260927 03:29:39.073810 11797 raft_consensus.cc:2454] T 249df48005ed40a6acad43f9e97fd6b7 P 2241c6bc25fb4d4c9a3141fe28f2cc37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15e0ef5da73245bba93caa21b1a15e2d in term 0.
I20260927 03:29:39.074124 12008 leader_election.cc:302] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15e0ef5da73245bba93caa21b1a15e2d, 2241c6bc25fb4d4c9a3141fe28f2cc37; no voters:
I20260927 03:29:39.074424 12148 raft_consensus.cc:2781] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:29:39.074479 12148 raft_consensus.cc:482] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:29:39.074501 12148 raft_consensus.cc:3037] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:29:39.075417 12148 raft_consensus.cc:504] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2241c6bc25fb4d4c9a3141fe28f2cc37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42559 } } peers { permanent_uuid: "15e0ef5da73245bba93caa21b1a15e2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42423 } } peers { permanent_uuid: "3081fca0a988479aa888dd8b771c04ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42051 } }
I20260927 03:29:39.075584 12148 leader_election.cc:288] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d [CANDIDATE]: Term 1 election: Requested vote from peers 2241c6bc25fb4d4c9a3141fe28f2cc37 (127.0.98.1:42559), 3081fca0a988479aa888dd8b771c04ea (127.0.98.2:42051)
I20260927 03:29:39.075776 11797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "249df48005ed40a6acad43f9e97fd6b7" candidate_uuid: "15e0ef5da73245bba93caa21b1a15e2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2241c6bc25fb4d4c9a3141fe28f2cc37"
I20260927 03:29:39.075814 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "249df48005ed40a6acad43f9e97fd6b7" candidate_uuid: "15e0ef5da73245bba93caa21b1a15e2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3081fca0a988479aa888dd8b771c04ea"
I20260927 03:29:39.075853 11797 raft_consensus.cc:3037] T 249df48005ed40a6acad43f9e97fd6b7 P 2241c6bc25fb4d4c9a3141fe28f2cc37 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:29:39.075886 11941 raft_consensus.cc:3037] T 249df48005ed40a6acad43f9e97fd6b7 P 3081fca0a988479aa888dd8b771c04ea [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:29:39.076712 11797 raft_consensus.cc:2454] T 249df48005ed40a6acad43f9e97fd6b7 P 2241c6bc25fb4d4c9a3141fe28f2cc37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15e0ef5da73245bba93caa21b1a15e2d in term 1.
I20260927 03:29:39.076766 11941 raft_consensus.cc:2454] T 249df48005ed40a6acad43f9e97fd6b7 P 3081fca0a988479aa888dd8b771c04ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15e0ef5da73245bba93caa21b1a15e2d in term 1.
I20260927 03:29:39.076915 12008 leader_election.cc:302] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15e0ef5da73245bba93caa21b1a15e2d, 3081fca0a988479aa888dd8b771c04ea; no voters:
I20260927 03:29:39.077039 12148 raft_consensus.cc:2781] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:29:39.077162 12148 raft_consensus.cc:686] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d [term 1 LEADER]: Becoming Leader. State: Replica: 15e0ef5da73245bba93caa21b1a15e2d, State: Running, Role: LEADER
I20260927 03:29:39.077239 12148 consensus_queue.cc:237] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2241c6bc25fb4d4c9a3141fe28f2cc37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42559 } } peers { permanent_uuid: "15e0ef5da73245bba93caa21b1a15e2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42423 } } peers { permanent_uuid: "3081fca0a988479aa888dd8b771c04ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42051 } }
I20260927 03:29:39.078066 11637 catalog_manager.cc:5495] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 15e0ef5da73245bba93caa21b1a15e2d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "15e0ef5da73245bba93caa21b1a15e2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2241c6bc25fb4d4c9a3141fe28f2cc37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15e0ef5da73245bba93caa21b1a15e2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3081fca0a988479aa888dd8b771c04ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42051 } health_report { overall_health: UNKNOWN } } }
I20260927 03:29:39.090377 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:29:39.087080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43976 (local address 127.0.98.3:42423)
0927 03:29:39.087124 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:29:39.087125 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:29:39.087132 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:29:39.087167 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:29:39.087168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:29:39.087190 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:29:39.087250 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:29:39.087414 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:29:39.088180 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:29:39.089279 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:29:39.089545 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:29:39.089609 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:29:39.089824 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:29:39.089965 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:29:39.090049 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 03:29:39.090278 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 03:29:39.090299 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:29:39.092466 11941 raft_consensus.cc:1261] T 249df48005ed40a6acad43f9e97fd6b7 P 3081fca0a988479aa888dd8b771c04ea [term 1 FOLLOWER]: Refusing update from remote peer 15e0ef5da73245bba93caa21b1a15e2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:29:39.092577 11797 raft_consensus.cc:1261] T 249df48005ed40a6acad43f9e97fd6b7 P 2241c6bc25fb4d4c9a3141fe28f2cc37 [term 1 FOLLOWER]: Refusing update from remote peer 15e0ef5da73245bba93caa21b1a15e2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:29:39.092906 12148 consensus_queue.cc:1035] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2241c6bc25fb4d4c9a3141fe28f2cc37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:29:39.093060 12150 consensus_queue.cc:1035] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3081fca0a988479aa888dd8b771c04ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:29:39.099795 12153 mvcc.cc:205] Tried to move back new op lower bound from 7333805175159988224 to 7333805175100944384. Current Snapshot: MvccSnapshot[applied={T|T < 7333805175159988224 or (T in {7333805175159988224})}]
I20260927 03:29:39.099797 11637 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:29:39.099895 11637 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:29:39.100625 11637 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7aab8afbce742298b1cd7680ddf40f5: Sending DeleteTablet for 3 replicas of tablet 249df48005ed40a6acad43f9e97fd6b7
I20260927 03:29:39.100986 12055 tablet_service.cc:1514] Processing DeleteTablet for tablet 249df48005ed40a6acad43f9e97fd6b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:29:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43970
I20260927 03:29:39.101080 11783 tablet_service.cc:1514] Processing DeleteTablet for tablet 249df48005ed40a6acad43f9e97fd6b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:29:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44802
I20260927 03:29:39.101300 12162 tablet_replica.cc:330] T 249df48005ed40a6acad43f9e97fd6b7 P 2241c6bc25fb4d4c9a3141fe28f2cc37: stopping tablet replica
I20260927 03:29:39.101449 12162 raft_consensus.cc:2229] T 249df48005ed40a6acad43f9e97fd6b7 P 2241c6bc25fb4d4c9a3141fe28f2cc37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:29:39.101457 11915 tablet_service.cc:1514] Processing DeleteTablet for tablet 249df48005ed40a6acad43f9e97fd6b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:29:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32900
I20260927 03:29:39.101576 12162 pending_rounds.cc:62] T 249df48005ed40a6acad43f9e97fd6b7 P 2241c6bc25fb4d4c9a3141fe28f2cc37: Trying to abort 2 pending ops.
I20260927 03:29:39.101611 12162 pending_rounds.cc:68] T 249df48005ed40a6acad43f9e97fd6b7 P 2241c6bc25fb4d4c9a3141fe28f2cc37: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7333805175159988224 op_type: WRITE_OP write_request { tablet_id: "249df48005ed40a6acad43f9e97fd6b7" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\317\235\342\325\006\"8\n\ntest-admin\022*\n 5c15235e72eb481891af106f66e4bcc1\020\001\030\001 \001(\001" signature: "<redacted>""d\324\304\237i\273[\004\223\251\213\272V\350,\0359;\322\333\361\215JX\t\010\220\321\364\257\317\r5\200\036\340\342TfF\300\377\366\307Z\037\023\243\375\274@\357\217nu|\215\300\322\221\361\001%c" signing_key_seq_num: 0 } } request_id { client_id: "51aba98ba632417496f51edb7a71c563" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260927 03:29:39.101763 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 11718
I20260927 03:29:39.101989 12162 pending_rounds.cc:68] T 249df48005ed40a6acad43f9e97fd6b7 P 2241c6bc25fb4d4c9a3141fe28f2cc37: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333805175100944384 op_type: NO_OP noop_request { }
I20260927 03:29:39.102067 12162 raft_consensus.cc:2866] T 249df48005ed40a6acad43f9e97fd6b7 P 2241c6bc25fb4d4c9a3141fe28f2cc37 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260927 03:29:39.102097 12162 raft_consensus.cc:2258] T 249df48005ed40a6acad43f9e97fd6b7 P 2241c6bc25fb4d4c9a3141fe28f2cc37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:29:39.103552 12161 tablet_replica.cc:330] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d: stopping tablet replica
I20260927 03:29:39.103650 12161 raft_consensus.cc:2229] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:29:39.103740 12161 raft_consensus.cc:2258] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:29:39.104280 12161 ts_tablet_manager.cc:1905] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:29:39.106155 12161 ts_tablet_manager.cc:1918] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:29:39.106220 12161 log.cc:1196] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790479675865103-392-0/minicluster-data/ts-2/wal/wals/249df48005ed40a6acad43f9e97fd6b7
I20260927 03:29:39.106495 12161 ts_tablet_manager.cc:1939] T 249df48005ed40a6acad43f9e97fd6b7 P 15e0ef5da73245bba93caa21b1a15e2d: Deleting consensus metadata
W20260927 03:29:39.109416 12008 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42559 (error 108)
I20260927 03:29:39.109561 11622 catalog_manager.cc:4841] TS 15e0ef5da73245bba93caa21b1a15e2d (127.0.98.3:42423): tablet 249df48005ed40a6acad43f9e97fd6b7 (table test-table [id=5c15235e72eb481891af106f66e4bcc1]) successfully deleted
W20260927 03:29:39.110276 11621 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42559 (error 108)
I20260927 03:29:39.110926 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:29:39.110800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42559 (local address 127.0.0.1:44816)
0927 03:29:39.110855 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 03:29:39.110888 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 03:29:39.111150 11621 catalog_manager.cc:4584] TS 2241c6bc25fb4d4c9a3141fe28f2cc37 (127.0.98.1:42559): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 249df48005ed40a6acad43f9e97fd6b7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42559: connect: Connection refused (error 111)
I20260927 03:29:39.111574 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 11857
W20260927 03:29:39.118506 12138 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:29:39.118153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42051 (local address 127.0.0.1:32916)
0927 03:29:39.118216 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 03:29:39.118230 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:29:39.118280 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:29:39.118444 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42051: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 03:29:39.119251 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 11992
I20260927 03:29:39.127293 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 11605
I20260927 03:29:39.133179 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 03:29:39.477993 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 03:29:39.478096 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 03:29:39.823284 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 03:29:39.823 UTC [11191] LOG: received smart shutdown request
2026-09-27 03:29:39.827 UTC [11191] LOG: background worker "logical replication launcher" (PID 11200) exited with exit code 1
2026-09-27 03:29:39.828 UTC [11195] LOG: shutting down
2026-09-27 03:29:40.001 UTC [11191] LOG: database system is shut down
2026-09-27T03:29:40Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46000 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:29:40 dist-test-slave-j0hf krb5kdc[12174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:29:40 dist-test-slave-j0hf krb5kdc[12174](info): set up 2 sockets
Sep 27 03:29:40 dist-test-slave-j0hf krb5kdc[12174](info): commencing operation
krb5kdc: starting...
W20260927 03:29:41.163807 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:29:41 dist-test-slave-j0hf krb5kdc[12174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:29:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:29:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 03:29:41.329874 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task7YYkzb/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/postgres -l logfile start
2026-09-27 03:29:43.806 UTC [12204] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 03:29:43.807 UTC [12204] LOG: listening on IPv4 address "127.0.98.20", port 39827
2026-09-27 03:29:43.810 UTC [12204] LOG: listening on Unix socket "/tmp/.s.PGSQL.39827"
2026-09-27 03:29:43.831 UTC [12207] LOG: database system was shut down at 2026-09-27 03:29:42 UTC
2026-09-27 03:29:43.835 UTC [12204] LOG: database system is ready to accept connections
I20260927 03:29:44.787552 392 mini_postgres.cc:96] Postgres bound to 39827
2026-09-27 03:29:44.793 UTC [12215] FATAL: database "slave" does not exist
127.0.98.20:39827 - accepting connections
I20260927 03:29:44.794199 392 mini_ranger.cc:161] Starting Ranger...
I20260927 03:29:44.808511 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 03:29:45.265128 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 03:29:45.265215 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ranger-admin
2026-09-27 03:29:45,530 [I] DB FLAVOR :POSTGRES
2026-09-27 03:29:45,530 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 03:29:45,530 [I] Checking connection..
2026-09-27 03:29:45,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 03:29:45,760 [I] Checking connection passed.
2026-09-27 03:29:45,760 [I] --------- Verifying version history table ---------
2026-09-27 03:29:45,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 03:29:46,010 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 03:29:46,010 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 03:29:46,242 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 03:29:46,242 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 03:29:46,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39827/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 03:29:46.483 UTC [12279] WARNING: there is no transaction in progress
2026-09-27 03:29:46,489 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 03:29:46,489 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 03:29:46,742 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 03:29:46,743 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 03:29:46,743 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 03:29:46,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 03:29:47,219 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j0hf.c.gcp-upstream.internal','N') ;"
2026-09-27 03:29:47,440 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 03:29:47,680 [I] Table x_portal_user does not exist in database ranger
2026-09-27 03:29:47,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 03:29:47,923 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 03:29:47,923 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 03:29:48,146 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 03:29:48,385 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 03:29:48,386 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39827/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 03:29:49.374 UTC [12387] WARNING: there is no transaction in progress
2026-09-27 03:29:49.390 UTC [12387] WARNING: there is no transaction in progress
2026-09-27 03:29:49.406 UTC [12387] WARNING: there is no transaction in progress
2026-09-27 03:29:49.422 UTC [12387] WARNING: there is no transaction in progress
2026-09-27 03:29:49.441 UTC [12387] WARNING: there is no transaction in progress
2026-09-27 03:29:49.459 UTC [12387] WARNING: there is no transaction in progress
2026-09-27 03:29:49.514 UTC [12387] WARNING: there is no transaction in progress
2026-09-27 03:29:49.521 UTC [12387] WARNING: there is no transaction in progress
2026-09-27 03:29:49.536 UTC [12387] WARNING: there is no transaction in progress
2026-09-27 03:29:49.546 UTC [12387] WARNING: there is no transaction in progress
2026-09-27 03:29:49.557 UTC [12387] WARNING: there is no transaction in progress
2026-09-27 03:29:49.571 UTC [12387] WARNING: there is no transaction in progress
2026-09-27 03:29:49.581 UTC [12387] WARNING: there is no transaction in progress
2026-09-27 03:29:49.587 UTC [12387] WARNING: there is no transaction in progress
2026-09-27 03:29:49.595 UTC [12387] WARNING: there is no transaction in progress
2026-09-27 03:29:49.603 UTC [12387] WARNING: there is no transaction in progress
2026-09-27 03:29:50,267 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 03:29:50,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 03:29:50,510 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 03:29:50,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 03:29:50,753 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 03:29:50,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 03:29:51,009 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 03:29:51,009 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 03:29:51,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 03:29:51,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j0hf.c.gcp-upstream.internal';"
2026-09-27 03:29:51,746 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 03:29:51,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 03:29:52,007 [I] DB_PATCHES have already been applied
I20260927 03:29:52.013281 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task7YYkzb/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 03:29:52.013381 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 03:29:52.016542 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:58315
Sep 27, 2026 3:29:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:29:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:29:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 3:29:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 3:29:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 3:29:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 3:29:52 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 03:29:53.152813 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.359s user 0.001s sys 0.008s
Sep 27, 2026 3:30:13 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 3:30:13 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 3:30:13 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 3:30:13 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 3:30:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 3:30:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 3:30:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 3:30:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 3:30:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 3:30:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 3:30:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 3:30:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 3:30:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 3:30:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 3:30:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 3:30:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 3:30:13 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 3:30:14 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 03:30:14.874624 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 03:30:14 dist-test-slave-j0hf krb5kdc[12174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479814, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 03:30:14.956945 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 03:30:14.965739 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 03:30:15.902629 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 03:30:15.902753 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ranger-kms
I20260927 03:30:15.902853 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:39827
2026-09-27 03:30:16,078 [I] DB FLAVOR :POSTGRES
2026-09-27 03:30:16,078 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 03:30:16,078 [I] Checking connection
2026-09-27 03:30:16,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39827/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 03:30:16,312 [I] connection success
2026-09-27 03:30:16,312 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 03:30:16,312 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 03:30:16,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39827/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 03:30:16,576 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 03:30:16,576 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 03:30:16,577 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 03:30:16,577 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39827/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 03:30:16,845 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 03:30:17.361408 392 mini_ranger_kms.cc:310] Created kms service
I20260927 03:30:17.549618 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 03:30:17.595079 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 03:30:17 dist-test-slave-j0hf krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479814, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 03:30:17.996047 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 03:30:17.996163 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task7YYkzb/build/release/bin/postgresql.jar:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task7YYkzb/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 03:30:17.996183 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 03:30:17.999985 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12567
I20260927 03:30:18.000067 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:46769
Sep 27, 2026 3:30:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:30:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:30:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 3:30:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 3:30:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 3:30:19 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 03:30:20.260677 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.304s user 0.000s sys 0.157s
I20260927 03:30:20.260790 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 03:30:20.260850 392 mini_ranger_kms.cc:401] 127.0.98.20:46769/kms/v1/keys
Sep 27 03:30:20 dist-test-slave-j0hf krb5kdc[12174](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790479820, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 03:30:20 dist-test-slave-j0hf krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479820, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 03:30:21 dist-test-slave-j0hf krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479820, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 03:30:23 dist-test-slave-j0hf krb5kdc[12174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:30:23.583069 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43395
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43395
--ranger_config_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46769/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:30:23.681625 12618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:30:23.681880 12618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:30:23.681937 12618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:30:23.683077 12618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:30:23.683151 12618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:30:23.683177 12618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:30:23.683207 12618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:30:23.683233 12618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:30:23.684875 12618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:46769/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43395
--ranger_config_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:30:23.685158 12618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:30:23.685406 12618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:30:23.690408 12623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:23.690459 12624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:23.690404 12626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:23.690693 12618 server_base.cc:1034] running on GCE node
I20260927 03:30:23.690923 12618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:30:23.691212 12618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:30:23.692407 12618 hybrid_clock.cc:648] HybridClock initialized: now 1790479823692357 us; error 65 us; skew 500 ppm
Sep 27 03:30:23 dist-test-slave-j0hf krb5kdc[12174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:30:23.695101 12618 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:30:23.695624 12618 webserver.cc:460] Webserver started at http://127.0.98.62:41557/ using document root <none> and password file <none>
I20260927 03:30:23.695850 12618 fs_manager.cc:359] Metadata directory not provided
I20260927 03:30:23.695912 12618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:30:23.696086 12618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:30:23 dist-test-slave-j0hf krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 03:30:23.747069 12618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "ada76c35bb424871a1903bdf9446ea02"
format_stamp: "Formatted at 2026-09-27 03:30:23 on dist-test-slave-j0hf"
server_key: "b9a59dff04494598d5009425947ff301"
server_key_iv: "1b7b0901ee1f495a039044ca8fa419e7"
server_key_version: "kuduclusterkey@0"
I20260927 03:30:23.747601 12618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "ada76c35bb424871a1903bdf9446ea02"
format_stamp: "Formatted at 2026-09-27 03:30:23 on dist-test-slave-j0hf"
server_key: "b9a59dff04494598d5009425947ff301"
server_key_iv: "1b7b0901ee1f495a039044ca8fa419e7"
server_key_version: "kuduclusterkey@0"
I20260927 03:30:23.749266 12618 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:30:23.763309 12633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:23.763761 12618 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:30:23.763880 12618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "ada76c35bb424871a1903bdf9446ea02"
format_stamp: "Formatted at 2026-09-27 03:30:23 on dist-test-slave-j0hf"
server_key: "b9a59dff04494598d5009425947ff301"
server_key_iv: "1b7b0901ee1f495a039044ca8fa419e7"
server_key_version: "kuduclusterkey@0"
I20260927 03:30:23.764021 12618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:30:23.779223 12618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:30:23.781404 12618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:30:23.781550 12618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:30:23.786439 12618 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43395
I20260927 03:30:23.786527 12685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43395 every 8 connection(s)
I20260927 03:30:23.786892 12618 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:30:23.787681 12686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:30:23.789983 12686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02: Bootstrap starting.
I20260927 03:30:23.790285 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 12618
I20260927 03:30:23.790390 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:30:23.790562 392 external_mini_cluster.cc:1428] Setting key 938fb7d52e636fb2ff2abe0fbe55d92b
I20260927 03:30:23.790704 12686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02: Neither blocks nor log segments found. Creating new log.
I20260927 03:30:23.791090 12686 log.cc:824] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02: Log is configured to *not* fsync() on all Append() calls
I20260927 03:30:23.791872 12686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02: No bootstrap required, opened a new log
I20260927 03:30:23.794090 12686 raft_consensus.cc:348] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada76c35bb424871a1903bdf9446ea02" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43395 } }
I20260927 03:30:23.794252 12686 raft_consensus.cc:374] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:30:23.794315 12686 raft_consensus.cc:729] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada76c35bb424871a1903bdf9446ea02, State: Initialized, Role: FOLLOWER
I20260927 03:30:23.794467 12686 consensus_queue.cc:260] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada76c35bb424871a1903bdf9446ea02" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43395 } }
I20260927 03:30:23.794577 12686 raft_consensus.cc:388] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:30:23.794631 12686 raft_consensus.cc:482] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:30:23.794703 12686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:30:23.795228 12686 raft_consensus.cc:504] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada76c35bb424871a1903bdf9446ea02" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43395 } }
I20260927 03:30:23.795353 12686 leader_election.cc:302] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ada76c35bb424871a1903bdf9446ea02; no voters:
I20260927 03:30:23.795539 12686 leader_election.cc:288] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:30:23.795567 12691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:30:23.795765 12691 raft_consensus.cc:686] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [term 1 LEADER]: Becoming Leader. State: Replica: ada76c35bb424871a1903bdf9446ea02, State: Running, Role: LEADER
I20260927 03:30:23.795877 12686 sys_catalog.cc:564] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:30:23.795888 12691 consensus_queue.cc:237] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada76c35bb424871a1903bdf9446ea02" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43395 } }
I20260927 03:30:23.796281 12691 sys_catalog.cc:455] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ada76c35bb424871a1903bdf9446ea02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada76c35bb424871a1903bdf9446ea02" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43395 } } }
I20260927 03:30:23.796351 12691 sys_catalog.cc:458] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [sys.catalog]: This master's current role is: LEADER
I20260927 03:30:23.796525 12693 sys_catalog.cc:455] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ada76c35bb424871a1903bdf9446ea02. Latest consensus state: current_term: 1 leader_uuid: "ada76c35bb424871a1903bdf9446ea02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada76c35bb424871a1903bdf9446ea02" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43395 } } }
I20260927 03:30:23.796639 12693 sys_catalog.cc:458] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [sys.catalog]: This master's current role is: LEADER
I20260927 03:30:23.796911 12695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:30:23.797631 12695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:30:23.798171 12686 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 27 03:30:23 dist-test-slave-j0hf krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:30:23.805290 12689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:23.791094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57904 (local address 127.0.98.62:43395)
0927 03:30:23.791269 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:23.791278 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:23.791342 (+ 64us) server_negotiation.cc:407] Connection header received
0927 03:30:23.791893 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:23.791902 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:23.791965 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:23.792113 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:23.792890 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:23.794271 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:23.795414 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:23.796612 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:23.797465 (+ 853us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:23.800336 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:23.800388 (+ 52us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:23.800388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:23.800524 (+ 136us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:23.803254 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:23.803688 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:23.803691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:23.803691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:23.803783 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:23.803971 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:23.803972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:23.803972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:23.804288 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:23.804869 (+ 581us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:23.804940 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 03:30:23.805085 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":68,"threads_started":1}
I20260927 03:30:23.810665 12695 catalog_manager.cc:1347] Generated new cluster ID: 3c98b919903b453d98116135f7123288
I20260927 03:30:23.810719 12695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:30:23.824184 12695 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:30:23.825018 12695 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:30:23.831183 12695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02: Generated new TSK 0
I20260927 03:30:23.831522 12695 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:30:24.620674 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46769/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43395
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:30:24.707832 12731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:30:24.708043 12731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:30:24.708066 12731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:30:24.709257 12731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:30:24.709301 12731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:30:24.709328 12731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:30:24.710788 12731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:46769/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:30:24.711007 12731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:30:24.711258 12731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:30:24.717350 12736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:24.717351 12737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:24.717393 12739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:24.717896 12731 server_base.cc:1034] running on GCE node
I20260927 03:30:24.718092 12731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:30:24.718339 12731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:30:24.719511 12731 hybrid_clock.cc:648] HybridClock initialized: now 1790479824719479 us; error 48 us; skew 500 ppm
Sep 27 03:30:24 dist-test-slave-j0hf krb5kdc[12174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:30:24.722591 12731 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:30:24.723112 12731 webserver.cc:460] Webserver started at http://127.0.98.1:46351/ using document root <none> and password file <none>
I20260927 03:30:24.723371 12731 fs_manager.cc:359] Metadata directory not provided
I20260927 03:30:24.723496 12731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:30:24.723670 12731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:30:24 dist-test-slave-j0hf krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 03:30:24.737496 12731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-0/data/instance:
uuid: "24a58ee4cf7e4326a8d0c129c0d68e9f"
format_stamp: "Formatted at 2026-09-27 03:30:24 on dist-test-slave-j0hf"
server_key: "0883a8b24f7b41bc688bff026f7b9093"
server_key_iv: "fdcacf70e56386971e43dc785235de16"
server_key_version: "kuduclusterkey@0"
I20260927 03:30:24.738011 12731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-0/wal/instance:
uuid: "24a58ee4cf7e4326a8d0c129c0d68e9f"
format_stamp: "Formatted at 2026-09-27 03:30:24 on dist-test-slave-j0hf"
server_key: "0883a8b24f7b41bc688bff026f7b9093"
server_key_iv: "fdcacf70e56386971e43dc785235de16"
server_key_version: "kuduclusterkey@0"
I20260927 03:30:24.743908 12731 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.002s
I20260927 03:30:24.763644 12746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:24.763856 12731 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.001s
I20260927 03:30:24.763965 12731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "24a58ee4cf7e4326a8d0c129c0d68e9f"
format_stamp: "Formatted at 2026-09-27 03:30:24 on dist-test-slave-j0hf"
server_key: "0883a8b24f7b41bc688bff026f7b9093"
server_key_iv: "fdcacf70e56386971e43dc785235de16"
server_key_version: "kuduclusterkey@0"
I20260927 03:30:24.764086 12731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:30:24.809229 12731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:30:24.811718 12731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:30:24.811865 12731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:30:24.812132 12731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:30:24.812474 12731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:30:24.812522 12731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:24.812554 12731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:30:24.812611 12731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:24.819193 12731 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41631
I20260927 03:30:24.819644 12731 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:30:24.819942 12861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41631 every 8 connection(s)
Sep 27 03:30:24 dist-test-slave-j0hf krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:30:24.828235 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 12731
I20260927 03:30:24.828347 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:30:24.828591 392 external_mini_cluster.cc:1428] Setting key 22a9829865516b9642a1d5284551bab9
I20260927 03:30:24.830924 12865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:24.820682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43395 (local address 127.0.98.1:50101)
0927 03:30:24.820968 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:24.821014 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:24.821673 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:24.822125 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:24.822129 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:24.822363 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:30:24.822593 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:24.822598 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:24.823642 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:24.823643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:24.824424 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:24.824426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:24.824935 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:24.824936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:24.825058 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:24.825627 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:30:24.825635 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:30:24.827152 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:30:24.829229 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:24.829236 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:24.829238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:24.829538 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:24.829772 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:24.829773 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:24.829774 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:24.829886 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:24.830216 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:30:24.830222 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:30:24.830394 (+ 172us) client_negotiation.cc:769] Sending connection context
0927 03:30:24.830515 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 03:30:24.830672 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":88,"threads_started":1}
I20260927 03:30:24.831250 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:24.820692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50101 (local address 127.0.98.62:43395)
0927 03:30:24.821871 (+ 1179us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:24.821873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:24.821882 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:30:24.821909 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:24.821910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:24.821923 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:24.821998 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:24.822712 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:24.823558 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:24.824528 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:24.824836 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:24.824917 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:24.827260 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:24.827262 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:24.827262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:24.827270 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:24.829130 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:24.829632 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:24.829633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:24.829634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:24.829702 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:24.829983 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:24.829984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:24.829985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:24.830122 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:24.830188 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:24.831063 (+ 875us) server_negotiation.cc:299] Negotiation successful
0927 03:30:24.831153 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1147,"thread_start_us":77,"threads_started":1}
I20260927 03:30:24.831898 12862 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43395
I20260927 03:30:24.831995 12862 heartbeater.cc:464] Registering TS with master...
I20260927 03:30:24.832175 12862 heartbeater.cc:511] Master 127.0.98.62:43395 requested a full tablet report, sending...
I20260927 03:30:24.832746 12650 ts_manager.cc:194] Registered new tserver with Master: 24a58ee4cf7e4326a8d0c129c0d68e9f (127.0.98.1:41631)
I20260927 03:30:24.833607 12650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50101
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:30:24.858635 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46769/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43395
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:30:24.954734 12868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:30:24.955081 12868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:30:24.955171 12868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:30:24.956940 12868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:30:24.957037 12868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:30:24.957118 12868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:30:24.959342 12868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:46769/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:30:24.959856 12868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:30:24.960145 12868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 27 03:30:24 dist-test-slave-j0hf krb5kdc[12174](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790479824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 03:30:24.965204 12875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:24.965210 12878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:24.965291 12876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:24.966336 12868 server_base.cc:1034] running on GCE node
I20260927 03:30:24.966643 12868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:30:24.966915 12868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:30:24.968071 12868 hybrid_clock.cc:648] HybridClock initialized: now 1790479824968057 us; error 24 us; skew 500 ppm
Sep 27 03:30:24 dist-test-slave-j0hf krb5kdc[12174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:30:24.971017 12868 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:30:24.971520 12868 webserver.cc:460] Webserver started at http://127.0.98.2:44843/ using document root <none> and password file <none>
I20260927 03:30:24.971781 12868 fs_manager.cc:359] Metadata directory not provided
I20260927 03:30:24.971833 12868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:30:24.971953 12868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:30:24 dist-test-slave-j0hf krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 03:30:24.986760 12868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-1/data/instance:
uuid: "b6197d818093464395353fd3f75c1c6f"
format_stamp: "Formatted at 2026-09-27 03:30:24 on dist-test-slave-j0hf"
server_key: "6a06bc106fafbe727b814594841d2d97"
server_key_iv: "b18b5b5aefe5b274b672450f6028fad6"
server_key_version: "kuduclusterkey@0"
I20260927 03:30:24.987193 12868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b6197d818093464395353fd3f75c1c6f"
format_stamp: "Formatted at 2026-09-27 03:30:24 on dist-test-slave-j0hf"
server_key: "6a06bc106fafbe727b814594841d2d97"
server_key_iv: "b18b5b5aefe5b274b672450f6028fad6"
server_key_version: "kuduclusterkey@0"
I20260927 03:30:24.991175 12868 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 03:30:25.014961 12885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:25.015183 12868 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.000s sys 0.001s
I20260927 03:30:25.015277 12868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "b6197d818093464395353fd3f75c1c6f"
format_stamp: "Formatted at 2026-09-27 03:30:24 on dist-test-slave-j0hf"
server_key: "6a06bc106fafbe727b814594841d2d97"
server_key_iv: "b18b5b5aefe5b274b672450f6028fad6"
server_key_version: "kuduclusterkey@0"
I20260927 03:30:25.015386 12868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:30:25.049793 12868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:30:25.052407 12868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:30:25.052556 12868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:30:25.052805 12868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:30:25.053201 12868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:30:25.053254 12868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:25.053290 12868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:30:25.053316 12868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:25.061759 12868 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42139
I20260927 03:30:25.062242 12868 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:30:25.062728 12999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42139 every 8 connection(s)
I20260927 03:30:25.065645 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 12868
I20260927 03:30:25.065754 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:30:25.065958 392 external_mini_cluster.cc:1428] Setting key 402c963a4585945851ab6fbeae3707bd
Sep 27 03:30:25 dist-test-slave-j0hf krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:30:25.074898 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:25.063538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34993 (local address 127.0.98.62:43395)
0927 03:30:25.063592 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:25.063594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:25.063958 (+ 364us) server_negotiation.cc:407] Connection header received
0927 03:30:25.064649 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:25.064650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:25.064664 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:25.064695 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:25.065508 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:25.066300 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:25.067977 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:25.068273 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:25.068318 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:25.071326 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:25.071328 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:25.071328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:25.071335 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:25.073206 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:25.073743 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:25.073745 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:25.073745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:25.073806 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:25.074104 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:25.074105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:25.074105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:25.074242 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:25.074288 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:25.074693 (+ 405us) server_negotiation.cc:299] Negotiation successful
0927 03:30:25.074781 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 03:30:25.075673 13002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:25.063312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43395 (local address 127.0.98.2:34993)
0927 03:30:25.063821 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:25.063861 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:25.064525 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:25.064874 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:25.064879 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:25.065125 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:30:25.065393 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:25.065399 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:25.066412 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:25.066413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:25.067832 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:25.067835 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:25.068419 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:25.068420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:25.068602 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:25.069242 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:30:25.069253 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:30:25.071210 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:30:25.073302 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:25.073308 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:25.073311 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:25.073608 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:25.073879 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:25.073880 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:25.073880 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:25.074024 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:25.074338 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:30:25.074345 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:30:25.074535 (+ 190us) client_negotiation.cc:769] Sending connection context
0927 03:30:25.075245 (+ 710us) client_negotiation.cc:240] Negotiation successful
0927 03:30:25.075412 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":81,"threads_started":1}
I20260927 03:30:25.076344 13000 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43395
I20260927 03:30:25.076437 13000 heartbeater.cc:464] Registering TS with master...
I20260927 03:30:25.076622 13000 heartbeater.cc:511] Master 127.0.98.62:43395 requested a full tablet report, sending...
I20260927 03:30:25.077051 12650 ts_manager.cc:194] Registered new tserver with Master: b6197d818093464395353fd3f75c1c6f (127.0.98.2:42139)
I20260927 03:30:25.077510 12650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34993
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:30:25.115175 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46769/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43395
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:30:25.229626 13005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:30:25.229890 13005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:30:25.229931 13005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:30:25.231274 13005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:30:25.231348 13005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:30:25.231395 13005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:30:25.233174 13005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:46769/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:30:25.233475 13005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:30:25.233758 13005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:30:25.239205 13013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:25.239205 13011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:25.239579 13010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:25.239637 13005 server_base.cc:1034] running on GCE node
I20260927 03:30:25.239876 13005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:30:25.240145 13005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:30:25.241298 13005 hybrid_clock.cc:648] HybridClock initialized: now 1790479825241264 us; error 59 us; skew 500 ppm
Sep 27 03:30:25 dist-test-slave-j0hf krb5kdc[12174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:30:25.251147 13005 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:30:25.251791 13005 webserver.cc:460] Webserver started at http://127.0.98.3:42259/ using document root <none> and password file <none>
I20260927 03:30:25.252053 13005 fs_manager.cc:359] Metadata directory not provided
I20260927 03:30:25.252100 13005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:30:25.252240 13005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:30:25 dist-test-slave-j0hf krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 03:30:25.272732 13005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-2/data/instance:
uuid: "573aa84f695143b2a80922f87923b787"
format_stamp: "Formatted at 2026-09-27 03:30:25 on dist-test-slave-j0hf"
server_key: "531726696e28c0e1f1e6a1a421679cfe"
server_key_iv: "538fca5d27c704383111a33beca229af"
server_key_version: "kuduclusterkey@0"
I20260927 03:30:25.273234 13005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-2/wal/instance:
uuid: "573aa84f695143b2a80922f87923b787"
format_stamp: "Formatted at 2026-09-27 03:30:25 on dist-test-slave-j0hf"
server_key: "531726696e28c0e1f1e6a1a421679cfe"
server_key_iv: "538fca5d27c704383111a33beca229af"
server_key_version: "kuduclusterkey@0"
I20260927 03:30:25.275313 13005 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260927 03:30:25.299612 13020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:25.299824 13005 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260927 03:30:25.299939 13005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "573aa84f695143b2a80922f87923b787"
format_stamp: "Formatted at 2026-09-27 03:30:25 on dist-test-slave-j0hf"
server_key: "531726696e28c0e1f1e6a1a421679cfe"
server_key_iv: "538fca5d27c704383111a33beca229af"
server_key_version: "kuduclusterkey@0"
I20260927 03:30:25.300061 13005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:30:25.311389 13005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:30:25.314071 13005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:30:25.314239 13005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:30:25.314467 13005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:30:25.314839 13005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:30:25.314889 13005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:25.314919 13005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:30:25.314947 13005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:25.322036 13005 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41129
I20260927 03:30:25.322515 13005 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:30:25.324606 13133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41129 every 8 connection(s)
I20260927 03:30:25.328584 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 13005
I20260927 03:30:25.328702 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:30:25.328949 392 external_mini_cluster.cc:1428] Setting key 793d0c434402eacbdbcc8b8e0b4db6d4
Sep 27 03:30:25 dist-test-slave-j0hf krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:30:25.337314 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:25.325811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55979 (local address 127.0.98.62:43395)
0927 03:30:25.325859 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:25.325861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:25.326199 (+ 338us) server_negotiation.cc:407] Connection header received
0927 03:30:25.327026 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:25.327027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:25.327041 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:25.327074 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:25.327998 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:25.328769 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:25.330646 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:25.330934 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:25.331023 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:25.333805 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:25.333807 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:25.333807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:25.333816 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:25.335664 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:25.336173 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:25.336174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:25.336175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:25.336236 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:25.336538 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:25.336538 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:25.336539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:25.336676 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:25.336722 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:25.337127 (+ 405us) server_negotiation.cc:299] Negotiation successful
0927 03:30:25.337214 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:30:25.338119 13136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:25.325514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43395 (local address 127.0.98.3:55979)
0927 03:30:25.326065 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:25.326111 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:25.326896 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:25.327276 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:25.327281 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:25.327569 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:30:25.327860 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:25.327866 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:25.328901 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:25.328902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:25.330537 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:25.330540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:25.331065 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:25.331066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:25.331210 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:25.331908 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:30:25.331927 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:30:25.333689 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:30:25.335762 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:25.335769 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:25.335772 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:25.336078 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:25.336313 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:25.336313 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:25.336314 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:25.336446 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:25.336772 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:30:25.336778 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:30:25.336963 (+ 185us) client_negotiation.cc:769] Sending connection context
0927 03:30:25.337709 (+ 746us) client_negotiation.cc:240] Negotiation successful
0927 03:30:25.337886 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":84,"threads_started":1}
I20260927 03:30:25.338590 13134 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43395
I20260927 03:30:25.338693 13134 heartbeater.cc:464] Registering TS with master...
I20260927 03:30:25.338888 13134 heartbeater.cc:511] Master 127.0.98.62:43395 requested a full tablet report, sending...
I20260927 03:30:25.339406 12650 ts_manager.cc:194] Registered new tserver with Master: 573aa84f695143b2a80922f87923b787 (127.0.98.3:41129)
I20260927 03:30:25.339934 12650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55979
I20260927 03:30:25.340852 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:30:25.341436 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 12731
I20260927 03:30:25.348023 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 12868
I20260927 03:30:25.352494 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 13005
I20260927 03:30:25.358925 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 12618
I20260927 03:30:25.363916 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43395
--webserver_interface=127.0.98.62
--webserver_port=41557
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43395
--ranger_config_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46769/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46769/kms with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:30:25.445336 13137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:30:25.445554 13137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:30:25.445575 13137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:30:25.446749 13137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:30:25.446794 13137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:30:25.446807 13137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:30:25.446820 13137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:30:25.446882 13137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:30:25.448591 13137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46769/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43395
--ranger_config_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:30:25.448848 13137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:30:25.449092 13137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:30:25.454035 13142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:25.454080 13143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:25.454103 13145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:25.454234 13137 server_base.cc:1034] running on GCE node
I20260927 03:30:25.454496 13137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:30:25.454802 13137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:30:25.455982 13137 hybrid_clock.cc:648] HybridClock initialized: now 1790479825455952 us; error 42 us; skew 500 ppm
Sep 27 03:30:25 dist-test-slave-j0hf krb5kdc[12174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:30:25.458554 13137 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:30:25.459035 13137 webserver.cc:460] Webserver started at http://127.0.98.62:41557/ using document root <none> and password file <none>
I20260927 03:30:25.459249 13137 fs_manager.cc:359] Metadata directory not provided
I20260927 03:30:25.459297 13137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 03:30:25 dist-test-slave-j0hf krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 03:30:25.520720 13137 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:30:25.521482 13153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:25.521636 13137 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:30:25.521718 13137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "ada76c35bb424871a1903bdf9446ea02"
format_stamp: "Formatted at 2026-09-27 03:30:23 on dist-test-slave-j0hf"
server_key: "b9a59dff04494598d5009425947ff301"
server_key_iv: "1b7b0901ee1f495a039044ca8fa419e7"
server_key_version: "kuduclusterkey@0"
I20260927 03:30:25.522099 13137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:30:25.532676 13137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:30:25.534325 13137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:30:25.534494 13137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:30:25.538898 13137 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43395
I20260927 03:30:25.538937 13205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43395 every 8 connection(s)
I20260927 03:30:25.539301 13137 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:30:25.539891 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 13137
I20260927 03:30:25.540199 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46769/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41631
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43395
--webserver_port=46351
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46769/kms with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:30:25.540761 13206 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:30:25.541585 13206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02: Bootstrap starting.
I20260927 03:30:25.552670 13206 log.cc:824] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02: Log is configured to *not* fsync() on all Append() calls
I20260927 03:30:25.554873 13206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:30:25.555197 13206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02: Bootstrap complete.
I20260927 03:30:25.558615 13206 raft_consensus.cc:348] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada76c35bb424871a1903bdf9446ea02" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43395 } }
I20260927 03:30:25.558959 13206 raft_consensus.cc:729] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada76c35bb424871a1903bdf9446ea02, State: Initialized, Role: FOLLOWER
I20260927 03:30:25.559134 13206 consensus_queue.cc:260] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada76c35bb424871a1903bdf9446ea02" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43395 } }
I20260927 03:30:25.559199 13206 raft_consensus.cc:388] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:30:25.559243 13206 raft_consensus.cc:482] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:30:25.559289 13206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [term 1 FOLLOWER]: Advancing to term 2
I20260927 03:30:25.560815 13206 raft_consensus.cc:504] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada76c35bb424871a1903bdf9446ea02" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43395 } }
I20260927 03:30:25.560930 13206 leader_election.cc:302] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ada76c35bb424871a1903bdf9446ea02; no voters:
I20260927 03:30:25.561101 13206 leader_election.cc:288] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 03:30:25.561199 13210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 03:30:25.561348 13206 sys_catalog.cc:564] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:30:25.561354 13210 raft_consensus.cc:686] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [term 2 LEADER]: Becoming Leader. State: Replica: ada76c35bb424871a1903bdf9446ea02, State: Running, Role: LEADER
I20260927 03:30:25.561573 13210 consensus_queue.cc:237] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada76c35bb424871a1903bdf9446ea02" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43395 } }
I20260927 03:30:25.561844 13210 sys_catalog.cc:455] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ada76c35bb424871a1903bdf9446ea02. Latest consensus state: current_term: 2 leader_uuid: "ada76c35bb424871a1903bdf9446ea02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada76c35bb424871a1903bdf9446ea02" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43395 } } }
I20260927 03:30:25.561937 13210 sys_catalog.cc:458] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [sys.catalog]: This master's current role is: LEADER
I20260927 03:30:25.561859 13212 sys_catalog.cc:455] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ada76c35bb424871a1903bdf9446ea02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada76c35bb424871a1903bdf9446ea02" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43395 } } }
I20260927 03:30:25.562021 13212 sys_catalog.cc:458] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02 [sys.catalog]: This master's current role is: LEADER
I20260927 03:30:25.562222 13218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:30:25.562932 13218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:30:25.563079 13218 catalog_manager.cc:1259] Loaded cluster ID: 3c98b919903b453d98116135f7123288
I20260927 03:30:25.563119 13218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:30:25.563287 13206 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 03:30:25.564157 13218 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:30:25.564416 13218 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02: Loaded TSK: 0
I20260927 03:30:25.564647 13218 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 03:30:25.662245 13208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:30:25.662506 13208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:30:25.662544 13208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:30:25.664386 13208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:30:25.664443 13208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:30:25.664480 13208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:30:25.666688 13208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46769/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41631
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46351
--tserver_master_addrs=127.0.98.62:43395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:30:25.666929 13208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:30:25.667182 13208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:30:25.673612 13246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:25.674256 13208 server_base.cc:1034] running on GCE node
W20260927 03:30:25.676179 13244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:25.676420 13243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:25.676772 13208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:30:25.677060 13208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:30:25.678243 13208 hybrid_clock.cc:648] HybridClock initialized: now 1790479825678214 us; error 45 us; skew 500 ppm
Sep 27 03:30:25 dist-test-slave-j0hf krb5kdc[12174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:30:25.681604 13208 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:30:25.682137 13208 webserver.cc:460] Webserver started at http://127.0.98.1:46351/ using document root <none> and password file <none>
I20260927 03:30:25.682391 13208 fs_manager.cc:359] Metadata directory not provided
I20260927 03:30:25.682477 13208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 03:30:25 dist-test-slave-j0hf krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 03:30:25.705832 13208 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 03:30:25.707564 13254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:25.707818 13208 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:30:25.707967 13208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "24a58ee4cf7e4326a8d0c129c0d68e9f"
format_stamp: "Formatted at 2026-09-27 03:30:24 on dist-test-slave-j0hf"
server_key: "0883a8b24f7b41bc688bff026f7b9093"
server_key_iv: "fdcacf70e56386971e43dc785235de16"
server_key_version: "kuduclusterkey@0"
I20260927 03:30:25.708491 13208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:30:25.745549 13208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:30:25.747852 13208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:30:25.748024 13208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:30:25.748281 13208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:30:25.748660 13208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:30:25.748713 13208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:25.748744 13208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:30:25.748764 13208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:25.756263 13208 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41631
I20260927 03:30:25.756433 13367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41631 every 8 connection(s)
I20260927 03:30:25.756754 13208 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:30:25.757211 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 13208
I20260927 03:30:25.757495 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46769/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42139
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43395
--webserver_port=44843
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46769/kms with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 03:30:25 dist-test-slave-j0hf krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:30:25.772385 13370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:25.757701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43395 (local address 127.0.98.1:60599)
0927 03:30:25.757877 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:25.757929 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:25.758698 (+ 769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:25.759563 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:25.759567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:25.759807 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:30:25.760115 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:25.760122 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:25.761453 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:25.761454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:25.762786 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:25.762789 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:25.763871 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:25.763872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:25.764048 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:25.764686 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:30:25.764697 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:30:25.766326 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:30:25.768927 (+ 2601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:25.768933 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:25.768936 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:25.769209 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:25.771056 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:25.771057 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:25.771058 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:25.771176 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:25.771651 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:30:25.771658 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:30:25.771814 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 03:30:25.771933 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 03:30:25.772103 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260927 03:30:25.772328 13371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:25.758103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60599 (local address 127.0.98.62:43395)
0927 03:30:25.759094 (+ 991us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:25.759101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:25.759131 (+ 30us) server_negotiation.cc:407] Connection header received
0927 03:30:25.759246 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:25.759252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:25.759313 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:25.760230 (+ 917us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:25.760315 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:25.761351 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:25.762890 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:25.763653 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:25.763761 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:25.766498 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:25.766503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:25.766503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:25.766519 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:25.768815 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:25.769466 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:25.769470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:25.769470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:25.770954 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:25.771252 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:25.771254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:25.771254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:25.771572 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:25.771672 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:25.771991 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 03:30:25.772105 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":939,"thread_start_us":81,"threads_started":1}
I20260927 03:30:25.773360 13368 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43395
I20260927 03:30:25.773470 13368 heartbeater.cc:464] Registering TS with master...
I20260927 03:30:25.773685 13368 heartbeater.cc:511] Master 127.0.98.62:43395 requested a full tablet report, sending...
W20260927 03:30:25.774534 13368 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43395 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 03:30:25.774616 13368 heartbeater.cc:464] Registering TS with master...
I20260927 03:30:25.774690 13368 heartbeater.cc:511] Master 127.0.98.62:43395 requested a full tablet report, sending...
I20260927 03:30:25.775034 13368 heartbeater.cc:464] Registering TS with master...
I20260927 03:30:25.775120 13368 heartbeater.cc:511] Master 127.0.98.62:43395 requested a full tablet report, sending...
W20260927 03:30:25.775453 13368 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 03:30:25.880870 13372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:30:25.881171 13372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:30:25.881220 13372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:30:25.882969 13372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:30:25.883034 13372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:30:25.883078 13372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:30:25.885303 13372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46769/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44843
--tserver_master_addrs=127.0.98.62:43395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:30:25.885560 13372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:30:25.885847 13372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:30:25.891237 13378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:25.891245 13377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:25.891815 13380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:25.892283 13372 server_base.cc:1034] running on GCE node
I20260927 03:30:25.892508 13372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:30:25.892796 13372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:30:25.893982 13372 hybrid_clock.cc:648] HybridClock initialized: now 1790479825893980 us; error 72 us; skew 500 ppm
Sep 27 03:30:25 dist-test-slave-j0hf krb5kdc[12174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:30:25.896924 13372 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:30:25.897470 13372 webserver.cc:460] Webserver started at http://127.0.98.2:44843/ using document root <none> and password file <none>
I20260927 03:30:25.897735 13372 fs_manager.cc:359] Metadata directory not provided
I20260927 03:30:25.897811 13372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 03:30:25 dist-test-slave-j0hf krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 03:30:25.919250 13372 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:30:25.920094 13388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:25.920312 13372 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:30:25.920425 13372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "b6197d818093464395353fd3f75c1c6f"
format_stamp: "Formatted at 2026-09-27 03:30:24 on dist-test-slave-j0hf"
server_key: "6a06bc106fafbe727b814594841d2d97"
server_key_iv: "b18b5b5aefe5b274b672450f6028fad6"
server_key_version: "kuduclusterkey@0"
I20260927 03:30:25.920902 13372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:30:25.953604 13372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:30:25.956015 13372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:30:25.956164 13372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:30:25.956473 13372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:30:25.956863 13372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:30:25.956909 13372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:25.956933 13372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:30:25.956946 13372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:25.963864 13372 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42139
I20260927 03:30:25.964438 13372 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:30:25.965750 13501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42139 every 8 connection(s)
I20260927 03:30:25.967089 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 13372
I20260927 03:30:25.967381 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46769/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41129
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43395
--webserver_port=42259
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46769/kms with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 03:30:25 dist-test-slave-j0hf krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:30:25.980911 13371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:25.966384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51993 (local address 127.0.98.62:43395)
0927 03:30:25.966432 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:25.966434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:25.966719 (+ 285us) server_negotiation.cc:407] Connection header received
0927 03:30:25.967491 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:25.967493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:25.967511 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:25.967545 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:25.968360 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:25.969157 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:25.970170 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:25.970489 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:25.970564 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:25.977169 (+ 6605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:25.977171 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:25.977171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:25.977182 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:25.979095 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:25.979683 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:25.979684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:25.979685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:25.979754 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:25.980094 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:25.980095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:25.980096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:25.980262 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:25.980340 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:25.980723 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 03:30:25.980808 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:30:25.981392 13504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:25.966409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43395 (local address 127.0.98.2:51993)
0927 03:30:25.966577 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:25.966619 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:25.967324 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:25.967728 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:25.967733 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:25.967983 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:30:25.968254 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:25.968260 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:25.969250 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:25.969251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:25.970079 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:25.970081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:25.970611 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:25.970612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:25.970775 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:25.975323 (+ 4548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:30:25.975334 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:30:25.977043 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:30:25.979196 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:25.979202 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:25.979205 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:25.979593 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:25.979836 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:25.979837 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:25.979837 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:25.980003 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:25.980341 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:30:25.980348 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:30:25.980542 (+ 194us) client_negotiation.cc:769] Sending connection context
0927 03:30:25.980674 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 03:30:25.980869 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
I20260927 03:30:25.982126 13502 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43395
I20260927 03:30:25.982251 13502 heartbeater.cc:464] Registering TS with master...
I20260927 03:30:25.982455 13502 heartbeater.cc:511] Master 127.0.98.62:43395 requested a full tablet report, sending...
W20260927 03:30:25.983139 13502 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43395 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 03:30:25.983219 13502 heartbeater.cc:464] Registering TS with master...
I20260927 03:30:25.983310 13502 heartbeater.cc:511] Master 127.0.98.62:43395 requested a full tablet report, sending...
I20260927 03:30:25.983651 13502 heartbeater.cc:464] Registering TS with master...
I20260927 03:30:25.983736 13502 heartbeater.cc:511] Master 127.0.98.62:43395 requested a full tablet report, sending...
W20260927 03:30:25.984093 13502 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 03:30:26.071095 13505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:30:26.071468 13505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:30:26.071563 13505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:30:26.073232 13505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:30:26.073300 13505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:30:26.073347 13505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:30:26.074880 13505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46769/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42259
--tserver_master_addrs=127.0.98.62:43395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:30:26.075161 13505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:30:26.075456 13505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:30:26.080736 13511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:26.081099 13510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:26.081337 13513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:26.081560 13505 server_base.cc:1034] running on GCE node
I20260927 03:30:26.081763 13505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:30:26.082032 13505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:30:26.083192 13505 hybrid_clock.cc:648] HybridClock initialized: now 1790479826083181 us; error 29 us; skew 500 ppm
Sep 27 03:30:26 dist-test-slave-j0hf krb5kdc[12174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:30:26.085927 13505 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:30:26.086410 13505 webserver.cc:460] Webserver started at http://127.0.98.3:42259/ using document root <none> and password file <none>
I20260927 03:30:26.086649 13505 fs_manager.cc:359] Metadata directory not provided
I20260927 03:30:26.086731 13505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 03:30:26 dist-test-slave-j0hf krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 03:30:26.114300 13505 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.001s
I20260927 03:30:26.115229 13522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:26.115489 13505 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:30:26.115576 13505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "573aa84f695143b2a80922f87923b787"
format_stamp: "Formatted at 2026-09-27 03:30:25 on dist-test-slave-j0hf"
server_key: "531726696e28c0e1f1e6a1a421679cfe"
server_key_iv: "538fca5d27c704383111a33beca229af"
server_key_version: "kuduclusterkey@0"
I20260927 03:30:26.116039 13505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:30:26.151060 13505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:30:26.153623 13505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:30:26.153793 13505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:30:26.154064 13505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:30:26.154518 13505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:30:26.154572 13505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:26.154609 13505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:30:26.154636 13505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:26.162009 13505 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41129
I20260927 03:30:26.162117 13635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41129 every 8 connection(s)
I20260927 03:30:26.162513 13505 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:30:26.168257 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 13505
Sep 27 03:30:26 dist-test-slave-j0hf krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:30:26.174254 13371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:26.163286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45837 (local address 127.0.98.62:43395)
0927 03:30:26.163340 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:26.163342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:26.163667 (+ 325us) server_negotiation.cc:407] Connection header received
0927 03:30:26.164200 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:26.164202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:26.164214 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:26.164272 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:26.164978 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:26.165744 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:26.167245 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:26.167506 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:26.167604 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:26.170250 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:26.170251 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:26.170252 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:26.170259 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:26.172130 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:26.172625 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:26.172627 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:26.172627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:26.172702 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:26.173429 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:26.173430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:26.173430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:26.173581 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:26.173625 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:26.174065 (+ 440us) server_negotiation.cc:299] Negotiation successful
0927 03:30:26.174159 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 03:30:26.175300 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:26.163279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43395 (local address 127.0.98.3:45837)
0927 03:30:26.163532 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:26.163572 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:26.164091 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:26.164349 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:26.164353 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:26.164580 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:30:26.164855 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:26.164863 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:26.165859 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:26.165860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:26.167157 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:26.167160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:26.167617 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:26.167618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:26.167748 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:26.168354 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:30:26.168367 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:30:26.170127 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:30:26.172221 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:26.172228 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:26.172230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:26.172541 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:26.173175 (+ 634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:26.173176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:26.173176 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:26.173341 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:26.173694 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:30:26.173701 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:30:26.173886 (+ 185us) client_negotiation.cc:769] Sending connection context
0927 03:30:26.174897 (+ 1011us) client_negotiation.cc:240] Negotiation successful
0927 03:30:26.175031 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":92,"threads_started":1}
I20260927 03:30:26.176043 13636 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43395
I20260927 03:30:26.176136 13636 heartbeater.cc:464] Registering TS with master...
I20260927 03:30:26.176321 13636 heartbeater.cc:511] Master 127.0.98.62:43395 requested a full tablet report, sending...
W20260927 03:30:26.177016 13636 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43395 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 03:30:26.177074 13636 heartbeater.cc:464] Registering TS with master...
I20260927 03:30:26.177137 13636 heartbeater.cc:511] Master 127.0.98.62:43395 requested a full tablet report, sending...
I20260927 03:30:26.177459 13636 heartbeater.cc:464] Registering TS with master...
I20260927 03:30:26.177532 13636 heartbeater.cc:511] Master 127.0.98.62:43395 requested a full tablet report, sending...
W20260927 03:30:26.177878 13636 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 03:30:26.181250 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:26.168655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57918 (local address 127.0.98.62:43395)
0927 03:30:26.168776 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:26.168779 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:26.168855 (+ 76us) server_negotiation.cc:407] Connection header received
0927 03:30:26.168935 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:26.168936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:26.168948 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:26.169009 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:26.170824 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:26.171533 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:26.172650 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:26.172957 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:26.173109 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:26.174638 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:26.174639 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:26.174640 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:26.174646 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:26.176744 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:26.179954 (+ 3210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:26.179955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:26.179955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:26.180016 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:26.180240 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:26.180241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:26.180242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:26.180352 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:26.181079 (+ 727us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:26.181116 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 03:30:26.181182 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":62,"threads_started":1}
Sep 27 03:30:26 dist-test-slave-j0hf krb5kdc[12174](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790479826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:30:26.775650 13368 heartbeater.cc:464] Registering TS with master...
I20260927 03:30:26.775805 13368 heartbeater.cc:511] Master 127.0.98.62:43395 requested a full tablet report, sending...
I20260927 03:30:26.776239 13170 ts_manager.cc:194] Registered new tserver with Master: 24a58ee4cf7e4326a8d0c129c0d68e9f (127.0.98.1:41631)
I20260927 03:30:26.777299 13170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60599
I20260927 03:30:26.984277 13502 heartbeater.cc:464] Registering TS with master...
I20260927 03:30:26.984426 13502 heartbeater.cc:511] Master 127.0.98.62:43395 requested a full tablet report, sending...
I20260927 03:30:26.984766 13170 ts_manager.cc:194] Registered new tserver with Master: b6197d818093464395353fd3f75c1c6f (127.0.98.2:42139)
I20260927 03:30:26.985309 13170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51993
Sep 27 03:30:27 dist-test-slave-j0hf krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 03:30:27.178090 13636 heartbeater.cc:464] Registering TS with master...
I20260927 03:30:27.178252 13636 heartbeater.cc:511] Master 127.0.98.62:43395 requested a full tablet report, sending...
I20260927 03:30:27.178617 13170 ts_manager.cc:194] Registered new tserver with Master: 573aa84f695143b2a80922f87923b787 (127.0.98.3:41129)
I20260927 03:30:27.179186 13170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45837
I20260927 03:30:27.185937 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:30:27.194708 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:27.186892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38328 (local address 127.0.98.62:43395)
0927 03:30:27.187167 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:27.187169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:27.187332 (+ 163us) server_negotiation.cc:407] Connection header received
0927 03:30:27.187354 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:27.187355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:27.187366 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:27.187416 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:27.188098 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:27.188861 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:27.189829 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:27.190111 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:27.190164 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:27.191635 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:27.191636 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:27.191637 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:27.191644 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:27.193320 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:27.193706 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:27.193707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:27.193707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:27.193761 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:27.193928 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:27.193928 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:27.193929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:27.194048 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:27.194131 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:27.194534 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 03:30:27.194612 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":53,"threads_started":1}
I20260927 03:30:27.197058 13170 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:30:27.199730 13170 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:30:27.212560 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:27.202884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41631 (local address 127.0.0.1:59112)
0927 03:30:27.207775 (+ 4891us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:27.207782 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:27.207810 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:27.208172 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:27.208175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:27.208177 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:30:27.208240 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:27.208242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:27.209718 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:27.209718 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:27.211279 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:27.211282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:27.212289 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:27.212290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:27.212359 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:27.212372 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 03:30:27.212432 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 03:30:27.212478 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4880,"thread_start_us":59,"threads_started":1}
I20260927 03:30:27.214030 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:27.203003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59112 (local address 127.0.98.1:41631)
0927 03:30:27.207925 (+ 4922us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:27.207931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:27.207941 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:30:27.207985 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:27.207987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:27.208002 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:27.208068 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:30:27.208555 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:27.209624 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:27.211373 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:27.212127 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:27.212200 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:27.212256 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:27.213888 (+ 1632us) server_negotiation.cc:299] Negotiation successful
0927 03:30:27.213940 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4886,"thread_start_us":49,"threads_started":1}
I20260927 03:30:27.214727 13302 tablet_service.cc:1467] Processing CreateTablet for tablet e1ebac187aa944d390d635b0a4673605 (DEFAULT_TABLE table=test-table [id=f8d138b3c33d450bbbee691a13ca992a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:30:27.215103 13302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1ebac187aa944d390d635b0a4673605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:30:27.215368 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:27.202567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41129 (local address 127.0.0.1:57630)
0927 03:30:27.207525 (+ 4958us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:27.207576 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:27.207657 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:27.211485 (+ 3828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:27.211486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:27.211488 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:30:27.211536 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:27.211538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:27.212874 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:27.212875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:27.214058 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:27.214061 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:27.215128 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:27.215129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:27.215189 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:27.215193 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:30:27.215245 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 03:30:27.215288 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4915,"thread_start_us":68,"threads_started":1}
I20260927 03:30:27.215708 13667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:27.203144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57630 (local address 127.0.98.3:41129)
0927 03:30:27.208353 (+ 5209us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:27.208357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:27.208365 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:30:27.208395 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:27.208397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:27.208411 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:27.208474 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:30:27.211642 (+ 3168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:27.212774 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:27.214162 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:27.214964 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:27.215036 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:27.215092 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:27.215588 (+ 496us) server_negotiation.cc:299] Negotiation successful
0927 03:30:27.215631 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5176,"thread_start_us":60,"threads_started":1}
I20260927 03:30:27.216302 13570 tablet_service.cc:1467] Processing CreateTablet for tablet e1ebac187aa944d390d635b0a4673605 (DEFAULT_TABLE table=test-table [id=f8d138b3c33d450bbbee691a13ca992a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:30:27.216640 13570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1ebac187aa944d390d635b0a4673605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:30:27.223107 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:27.202749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42139 (local address 127.0.0.1:52636)
0927 03:30:27.207658 (+ 4909us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:27.207668 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:27.207709 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:27.213055 (+ 5346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:27.213056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:27.213057 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:30:27.213095 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:27.213097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:27.220443 (+ 7346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:27.220444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:27.221582 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:27.221586 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:27.222545 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:27.222546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:27.222614 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:27.222620 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 03:30:27.222653 (+ 33us) client_negotiation.cc:240] Negotiation successful
0927 03:30:27.222693 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4891,"thread_start_us":43,"threads_started":1}
I20260927 03:30:27.223553 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:27.203297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52636 (local address 127.0.98.2:42139)
0927 03:30:27.212828 (+ 9531us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:27.212833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:27.212843 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:30:27.212876 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:27.212878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:27.212893 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:27.212966 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:30:27.213180 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:27.220305 (+ 7125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:27.221694 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:27.222437 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:27.223297 (+ 860us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:27.223390 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:27.223416 (+ 26us) server_negotiation.cc:299] Negotiation successful
0927 03:30:27.223478 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9496,"thread_start_us":68,"threads_started":1}
I20260927 03:30:27.224175 13436 tablet_service.cc:1467] Processing CreateTablet for tablet e1ebac187aa944d390d635b0a4673605 (DEFAULT_TABLE table=test-table [id=f8d138b3c33d450bbbee691a13ca992a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:30:27.224529 13436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1ebac187aa944d390d635b0a4673605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:30:27.224731 13669 tablet_bootstrap.cc:492] T e1ebac187aa944d390d635b0a4673605 P 24a58ee4cf7e4326a8d0c129c0d68e9f: Bootstrap starting.
I20260927 03:30:27.225384 13669 tablet_bootstrap.cc:654] T e1ebac187aa944d390d635b0a4673605 P 24a58ee4cf7e4326a8d0c129c0d68e9f: Neither blocks nor log segments found. Creating new log.
I20260927 03:30:27.227594 13670 tablet_bootstrap.cc:492] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787: Bootstrap starting.
I20260927 03:30:27.228277 13670 tablet_bootstrap.cc:654] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787: Neither blocks nor log segments found. Creating new log.
I20260927 03:30:27.228628 13670 log.cc:824] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787: Log is configured to *not* fsync() on all Append() calls
I20260927 03:30:27.229456 13670 tablet_bootstrap.cc:492] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787: No bootstrap required, opened a new log
I20260927 03:30:27.229533 13670 ts_tablet_manager.cc:1397] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:30:27.232167 13670 raft_consensus.cc:348] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573aa84f695143b2a80922f87923b787" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41129 } } peers { permanent_uuid: "b6197d818093464395353fd3f75c1c6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } } peers { permanent_uuid: "24a58ee4cf7e4326a8d0c129c0d68e9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41631 } }
I20260927 03:30:27.232348 13670 raft_consensus.cc:374] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:30:27.232398 13670 raft_consensus.cc:729] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 573aa84f695143b2a80922f87923b787, State: Initialized, Role: FOLLOWER
I20260927 03:30:27.232553 13670 consensus_queue.cc:260] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573aa84f695143b2a80922f87923b787" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41129 } } peers { permanent_uuid: "b6197d818093464395353fd3f75c1c6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } } peers { permanent_uuid: "24a58ee4cf7e4326a8d0c129c0d68e9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41631 } }
I20260927 03:30:27.232832 13670 ts_tablet_manager.cc:1428] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260927 03:30:27.232962 13636 heartbeater.cc:503] Master 127.0.98.62:43395 was elected leader, sending a full tablet report...
I20260927 03:30:27.235584 13669 log.cc:824] T e1ebac187aa944d390d635b0a4673605 P 24a58ee4cf7e4326a8d0c129c0d68e9f: Log is configured to *not* fsync() on all Append() calls
I20260927 03:30:27.235641 13671 tablet_bootstrap.cc:492] T e1ebac187aa944d390d635b0a4673605 P b6197d818093464395353fd3f75c1c6f: Bootstrap starting.
I20260927 03:30:27.236330 13671 tablet_bootstrap.cc:654] T e1ebac187aa944d390d635b0a4673605 P b6197d818093464395353fd3f75c1c6f: Neither blocks nor log segments found. Creating new log.
I20260927 03:30:27.236665 13671 log.cc:824] T e1ebac187aa944d390d635b0a4673605 P b6197d818093464395353fd3f75c1c6f: Log is configured to *not* fsync() on all Append() calls
I20260927 03:30:27.237772 13669 tablet_bootstrap.cc:492] T e1ebac187aa944d390d635b0a4673605 P 24a58ee4cf7e4326a8d0c129c0d68e9f: No bootstrap required, opened a new log
I20260927 03:30:27.237859 13669 ts_tablet_manager.cc:1397] T e1ebac187aa944d390d635b0a4673605 P 24a58ee4cf7e4326a8d0c129c0d68e9f: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20260927 03:30:27.237924 13671 tablet_bootstrap.cc:492] T e1ebac187aa944d390d635b0a4673605 P b6197d818093464395353fd3f75c1c6f: No bootstrap required, opened a new log
I20260927 03:30:27.237993 13671 ts_tablet_manager.cc:1397] T e1ebac187aa944d390d635b0a4673605 P b6197d818093464395353fd3f75c1c6f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:30:27.241144 13671 raft_consensus.cc:348] T e1ebac187aa944d390d635b0a4673605 P b6197d818093464395353fd3f75c1c6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573aa84f695143b2a80922f87923b787" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41129 } } peers { permanent_uuid: "b6197d818093464395353fd3f75c1c6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } } peers { permanent_uuid: "24a58ee4cf7e4326a8d0c129c0d68e9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41631 } }
I20260927 03:30:27.241144 13669 raft_consensus.cc:348] T e1ebac187aa944d390d635b0a4673605 P 24a58ee4cf7e4326a8d0c129c0d68e9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573aa84f695143b2a80922f87923b787" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41129 } } peers { permanent_uuid: "b6197d818093464395353fd3f75c1c6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } } peers { permanent_uuid: "24a58ee4cf7e4326a8d0c129c0d68e9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41631 } }
I20260927 03:30:27.241325 13669 raft_consensus.cc:374] T e1ebac187aa944d390d635b0a4673605 P 24a58ee4cf7e4326a8d0c129c0d68e9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:30:27.241325 13671 raft_consensus.cc:374] T e1ebac187aa944d390d635b0a4673605 P b6197d818093464395353fd3f75c1c6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:30:27.241369 13669 raft_consensus.cc:729] T e1ebac187aa944d390d635b0a4673605 P 24a58ee4cf7e4326a8d0c129c0d68e9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a58ee4cf7e4326a8d0c129c0d68e9f, State: Initialized, Role: FOLLOWER
I20260927 03:30:27.241369 13671 raft_consensus.cc:729] T e1ebac187aa944d390d635b0a4673605 P b6197d818093464395353fd3f75c1c6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6197d818093464395353fd3f75c1c6f, State: Initialized, Role: FOLLOWER
I20260927 03:30:27.241520 13671 consensus_queue.cc:260] T e1ebac187aa944d390d635b0a4673605 P b6197d818093464395353fd3f75c1c6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573aa84f695143b2a80922f87923b787" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41129 } } peers { permanent_uuid: "b6197d818093464395353fd3f75c1c6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } } peers { permanent_uuid: "24a58ee4cf7e4326a8d0c129c0d68e9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41631 } }
I20260927 03:30:27.241520 13669 consensus_queue.cc:260] T e1ebac187aa944d390d635b0a4673605 P 24a58ee4cf7e4326a8d0c129c0d68e9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573aa84f695143b2a80922f87923b787" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41129 } } peers { permanent_uuid: "b6197d818093464395353fd3f75c1c6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } } peers { permanent_uuid: "24a58ee4cf7e4326a8d0c129c0d68e9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41631 } }
I20260927 03:30:27.241777 13669 ts_tablet_manager.cc:1428] T e1ebac187aa944d390d635b0a4673605 P 24a58ee4cf7e4326a8d0c129c0d68e9f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260927 03:30:27.241777 13671 ts_tablet_manager.cc:1428] T e1ebac187aa944d390d635b0a4673605 P b6197d818093464395353fd3f75c1c6f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260927 03:30:27.241899 13368 heartbeater.cc:503] Master 127.0.98.62:43395 was elected leader, sending a full tablet report...
I20260927 03:30:27.241899 13502 heartbeater.cc:503] Master 127.0.98.62:43395 was elected leader, sending a full tablet report...
W20260927 03:30:27.259922 13369 tablet.cc:2369] T e1ebac187aa944d390d635b0a4673605 P 24a58ee4cf7e4326a8d0c129c0d68e9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 03:30:27 dist-test-slave-j0hf krb5kdc[12174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 03:30:27.380676 13673 raft_consensus.cc:482] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:30:27.380852 13673 raft_consensus.cc:504] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573aa84f695143b2a80922f87923b787" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41129 } } peers { permanent_uuid: "b6197d818093464395353fd3f75c1c6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } } peers { permanent_uuid: "24a58ee4cf7e4326a8d0c129c0d68e9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41631 } }
I20260927 03:30:27.381325 13673 leader_election.cc:288] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6197d818093464395353fd3f75c1c6f (127.0.98.2:42139), 24a58ee4cf7e4326a8d0c129c0d68e9f (127.0.98.1:41631)
I20260927 03:30:27.386176 13677 raft_consensus.cc:482] T e1ebac187aa944d390d635b0a4673605 P b6197d818093464395353fd3f75c1c6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:30:27.386325 13677 raft_consensus.cc:504] T e1ebac187aa944d390d635b0a4673605 P b6197d818093464395353fd3f75c1c6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573aa84f695143b2a80922f87923b787" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41129 } } peers { permanent_uuid: "b6197d818093464395353fd3f75c1c6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } } peers { permanent_uuid: "24a58ee4cf7e4326a8d0c129c0d68e9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41631 } }
I20260927 03:30:27.386825 13677 leader_election.cc:288] T e1ebac187aa944d390d635b0a4673605 P b6197d818093464395353fd3f75c1c6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 573aa84f695143b2a80922f87923b787 (127.0.98.3:41129), 24a58ee4cf7e4326a8d0c129c0d68e9f (127.0.98.1:41631)
I20260927 03:30:27.387838 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:27.382345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45759 (local address 127.0.98.2:42139)
0927 03:30:27.382402 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:27.382405 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:27.382997 (+ 592us) server_negotiation.cc:407] Connection header received
0927 03:30:27.383113 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:27.383115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:27.383125 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:27.383156 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:30:27.383395 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:27.384253 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:27.385720 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:27.386299 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:27.386340 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:27.386352 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:27.387721 (+ 1369us) server_negotiation.cc:299] Negotiation successful
0927 03:30:27.387770 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 03:30:27.388296 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:27.381469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42139 (local address 127.0.98.3:45759)
0927 03:30:27.382908 (+ 1439us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:27.382922 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:27.383030 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:27.383225 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:27.383226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:27.383228 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:30:27.383316 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:27.383318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:27.384361 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:27.384362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:27.385635 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:27.385636 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:27.387016 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:27.387017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:27.387084 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:27.387088 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:30:27.388124 (+ 1036us) client_negotiation.cc:240] Negotiation successful
0927 03:30:27.388183 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1077,"thread_start_us":104,"threads_started":1}
I20260927 03:30:27.388556 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:27.382390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39901 (local address 127.0.98.1:41631)
0927 03:30:27.382432 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:27.382434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:27.385094 (+ 2660us) server_negotiation.cc:407] Connection header received
0927 03:30:27.385192 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:27.385193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:27.385206 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:27.385234 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:30:27.385428 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:27.386100 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:27.387378 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:27.387998 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:27.388044 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:27.388053 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:27.388424 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 03:30:27.388466 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:30:27.388756 13679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:27.381698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41631 (local address 127.0.98.3:39901)
0927 03:30:27.385019 (+ 3321us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:27.385031 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:27.385120 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:27.385294 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:27.385295 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:27.385296 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:30:27.385354 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:27.385355 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:27.386195 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:27.386195 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:27.387285 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:27.387287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:27.388259 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:27.388260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:27.388332 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:27.388336 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:30:27.388643 (+ 307us) client_negotiation.cc:240] Negotiation successful
0927 03:30:27.388688 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3301,"thread_start_us":526,"threads_started":1}
I20260927 03:30:27.388792 13456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1ebac187aa944d390d635b0a4673605" candidate_uuid: "573aa84f695143b2a80922f87923b787" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6197d818093464395353fd3f75c1c6f" is_pre_election: true
I20260927 03:30:27.388901 13456 raft_consensus.cc:2454] T e1ebac187aa944d390d635b0a4673605 P b6197d818093464395353fd3f75c1c6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 573aa84f695143b2a80922f87923b787 in term 0.
I20260927 03:30:27.389179 13322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1ebac187aa944d390d635b0a4673605" candidate_uuid: "573aa84f695143b2a80922f87923b787" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a58ee4cf7e4326a8d0c129c0d68e9f" is_pre_election: true
I20260927 03:30:27.389314 13322 raft_consensus.cc:2454] T e1ebac187aa944d390d635b0a4673605 P 24a58ee4cf7e4326a8d0c129c0d68e9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 573aa84f695143b2a80922f87923b787 in term 0.
I20260927 03:30:27.389465 13523 leader_election.cc:302] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 573aa84f695143b2a80922f87923b787, b6197d818093464395353fd3f75c1c6f; no voters:
I20260927 03:30:27.389582 13673 raft_consensus.cc:2781] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:30:27.389627 13673 raft_consensus.cc:482] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:30:27.389658 13673 raft_consensus.cc:3037] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:30:27.390358 13673 raft_consensus.cc:504] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573aa84f695143b2a80922f87923b787" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41129 } } peers { permanent_uuid: "b6197d818093464395353fd3f75c1c6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } } peers { permanent_uuid: "24a58ee4cf7e4326a8d0c129c0d68e9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41631 } }
I20260927 03:30:27.390515 13673 leader_election.cc:288] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787 [CANDIDATE]: Term 1 election: Requested vote from peers b6197d818093464395353fd3f75c1c6f (127.0.98.2:42139), 24a58ee4cf7e4326a8d0c129c0d68e9f (127.0.98.1:41631)
I20260927 03:30:27.390676 13322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1ebac187aa944d390d635b0a4673605" candidate_uuid: "573aa84f695143b2a80922f87923b787" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a58ee4cf7e4326a8d0c129c0d68e9f"
I20260927 03:30:27.390676 13456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1ebac187aa944d390d635b0a4673605" candidate_uuid: "573aa84f695143b2a80922f87923b787" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6197d818093464395353fd3f75c1c6f"
I20260927 03:30:27.390741 13456 raft_consensus.cc:3037] T e1ebac187aa944d390d635b0a4673605 P b6197d818093464395353fd3f75c1c6f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:30:27.390743 13322 raft_consensus.cc:3037] T e1ebac187aa944d390d635b0a4673605 P 24a58ee4cf7e4326a8d0c129c0d68e9f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:30:27.391600 13322 raft_consensus.cc:2454] T e1ebac187aa944d390d635b0a4673605 P 24a58ee4cf7e4326a8d0c129c0d68e9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 573aa84f695143b2a80922f87923b787 in term 1.
I20260927 03:30:27.391600 13456 raft_consensus.cc:2454] T e1ebac187aa944d390d635b0a4673605 P b6197d818093464395353fd3f75c1c6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 573aa84f695143b2a80922f87923b787 in term 1.
I20260927 03:30:27.391780 13525 leader_election.cc:302] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24a58ee4cf7e4326a8d0c129c0d68e9f, 573aa84f695143b2a80922f87923b787; no voters:
I20260927 03:30:27.391855 13673 raft_consensus.cc:2781] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:30:27.391980 13673 raft_consensus.cc:686] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787 [term 1 LEADER]: Becoming Leader. State: Replica: 573aa84f695143b2a80922f87923b787, State: Running, Role: LEADER
I20260927 03:30:27.392081 13673 consensus_queue.cc:237] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573aa84f695143b2a80922f87923b787" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41129 } } peers { permanent_uuid: "b6197d818093464395353fd3f75c1c6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } } peers { permanent_uuid: "24a58ee4cf7e4326a8d0c129c0d68e9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41631 } }
I20260927 03:30:27.392825 13170 catalog_manager.cc:5495] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787 reported cstate change: term changed from 0 to 1, leader changed from <none> to 573aa84f695143b2a80922f87923b787 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "573aa84f695143b2a80922f87923b787" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573aa84f695143b2a80922f87923b787" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6197d818093464395353fd3f75c1c6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24a58ee4cf7e4326a8d0c129c0d68e9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41631 } health_report { overall_health: UNKNOWN } } }
I20260927 03:30:27.396495 13667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:27.387581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39715 (local address 127.0.98.3:41129)
0927 03:30:27.387623 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:27.387625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:27.393295 (+ 5670us) server_negotiation.cc:407] Connection header received
0927 03:30:27.393396 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:27.393396 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:27.393406 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:27.393437 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:30:27.393683 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:27.394260 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:27.395588 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:27.396133 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:27.396175 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:27.396185 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:27.396386 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 03:30:27.396420 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:30:27.396746 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:27.387091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41129 (local address 127.0.98.2:39715)
0927 03:30:27.393228 (+ 6137us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:27.393241 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:27.393323 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:27.393510 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:27.393511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:27.393512 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:30:27.393588 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:27.393590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:27.394356 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:27.394357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:27.395469 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:27.395473 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:27.396241 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:27.396241 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:27.396313 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:27.396317 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:30:27.396602 (+ 285us) client_negotiation.cc:240] Negotiation successful
0927 03:30:27.396654 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6115,"thread_start_us":97,"threads_started":1}
I20260927 03:30:27.397013 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1ebac187aa944d390d635b0a4673605" candidate_uuid: "b6197d818093464395353fd3f75c1c6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "573aa84f695143b2a80922f87923b787" is_pre_election: true
I20260927 03:30:27.399174 13681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:27.387317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41631 (local address 127.0.98.2:40343)
0927 03:30:27.395527 (+ 8210us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:27.395539 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:27.395614 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:27.395775 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:27.395776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:27.395777 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:30:27.395832 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:27.395833 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:27.396942 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:27.396943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:27.398137 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:27.398139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:27.398892 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:27.398894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:27.398980 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:27.398985 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:30:27.399052 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 03:30:27.399090 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8190,"thread_start_us":78,"threads_started":1}
I20260927 03:30:27.399194 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:27.388090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40343 (local address 127.0.98.1:41631)
0927 03:30:27.388627 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:27.388629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:27.395589 (+ 6960us) server_negotiation.cc:407] Connection header received
0927 03:30:27.395683 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:27.395684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:27.395693 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:27.395721 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:30:27.395895 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:27.396561 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:27.398232 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:27.398765 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:27.398853 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:27.398865 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:27.399074 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 03:30:27.399123 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":107,"threads_started":1}
I20260927 03:30:27.399389 13322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1ebac187aa944d390d635b0a4673605" candidate_uuid: "b6197d818093464395353fd3f75c1c6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a58ee4cf7e4326a8d0c129c0d68e9f" is_pre_election: true
I20260927 03:30:27.399537 13322 raft_consensus.cc:2379] T e1ebac187aa944d390d635b0a4673605 P 24a58ee4cf7e4326a8d0c129c0d68e9f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b6197d818093464395353fd3f75c1c6f in current term 1: Already voted for candidate 573aa84f695143b2a80922f87923b787 in this term.
I20260927 03:30:27.399824 13391 leader_election.cc:302] T e1ebac187aa944d390d635b0a4673605 P b6197d818093464395353fd3f75c1c6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b6197d818093464395353fd3f75c1c6f; no voters: 24a58ee4cf7e4326a8d0c129c0d68e9f, 573aa84f695143b2a80922f87923b787
I20260927 03:30:27.399977 13677 raft_consensus.cc:2726] T e1ebac187aa944d390d635b0a4673605 P b6197d818093464395353fd3f75c1c6f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260927 03:30:27.413839 13637 tablet.cc:2369] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:30:27.433804 13667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:27.430322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57638 (local address 127.0.98.3:41129)
0927 03:30:27.430369 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:27.430371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:27.430380 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:30:27.430427 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:27.430428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:27.430453 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:27.430527 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:30:27.430745 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:27.431533 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:27.432635 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:27.432934 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:27.432995 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:27.433206 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:30:27.433348 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:30:27.433435 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:27.433695 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 03:30:27.433719 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 03:30:27.436496 13322 raft_consensus.cc:1261] T e1ebac187aa944d390d635b0a4673605 P 24a58ee4cf7e4326a8d0c129c0d68e9f [term 1 FOLLOWER]: Refusing update from remote peer 573aa84f695143b2a80922f87923b787: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:30:27.436828 13673 consensus_queue.cc:1035] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24a58ee4cf7e4326a8d0c129c0d68e9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:30:27.439105 13456 raft_consensus.cc:1261] T e1ebac187aa944d390d635b0a4673605 P b6197d818093464395353fd3f75c1c6f [term 1 FOLLOWER]: Refusing update from remote peer 573aa84f695143b2a80922f87923b787: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:30:27.439368 13673 consensus_queue.cc:1035] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6197d818093464395353fd3f75c1c6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:30:27.444710 13688 mvcc.cc:205] Tried to move back new op lower bound from 7333805373176737792 to 7333805372998270976. Current Snapshot: MvccSnapshot[applied={T|T < 7333805373176737792}]
I20260927 03:30:27.444844 13691 mvcc.cc:205] Tried to move back new op lower bound from 7333805373176737792 to 7333805372998270976. Current Snapshot: MvccSnapshot[applied={T|T < 7333805373176737792}]
I20260927 03:30:27.449304 13170 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:30:27.449409 13170 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:30:27.450336 13170 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ada76c35bb424871a1903bdf9446ea02: Sending DeleteTablet for 3 replicas of tablet e1ebac187aa944d390d635b0a4673605
I20260927 03:30:27.450472 13689 mvcc.cc:205] Tried to move back new op lower bound from 7333805373176737792 to 7333805372998270976. Current Snapshot: MvccSnapshot[applied={T|T < 7333805373176737792}]
I20260927 03:30:27.450874 13570 tablet_service.cc:1514] Processing DeleteTablet for tablet e1ebac187aa944d390d635b0a4673605 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:30:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57630
I20260927 03:30:27.451112 13436 tablet_service.cc:1514] Processing DeleteTablet for tablet e1ebac187aa944d390d635b0a4673605 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:30:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52636
I20260927 03:30:27.451812 13302 tablet_service.cc:1514] Processing DeleteTablet for tablet e1ebac187aa944d390d635b0a4673605 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:30:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59112
I20260927 03:30:27.452374 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 13208
I20260927 03:30:27.459537 13698 tablet_replica.cc:330] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787: stopping tablet replica
I20260927 03:30:27.459669 13698 raft_consensus.cc:2229] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:30:27.459818 13698 raft_consensus.cc:2258] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:30:27.460372 13698 ts_tablet_manager.cc:1905] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:30:27.460986 13699 tablet_replica.cc:330] T e1ebac187aa944d390d635b0a4673605 P b6197d818093464395353fd3f75c1c6f: stopping tablet replica
I20260927 03:30:27.461109 13699 raft_consensus.cc:2229] T e1ebac187aa944d390d635b0a4673605 P b6197d818093464395353fd3f75c1c6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:30:27.461280 13699 raft_consensus.cc:2258] T e1ebac187aa944d390d635b0a4673605 P b6197d818093464395353fd3f75c1c6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:30:27.462011 13699 ts_tablet_manager.cc:1905] T e1ebac187aa944d390d635b0a4673605 P b6197d818093464395353fd3f75c1c6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:30:27.464051 13698 ts_tablet_manager.cc:1918] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:30:27.464131 13698 log.cc:1196] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-2/wal/wals/e1ebac187aa944d390d635b0a4673605
I20260927 03:30:27.464403 13698 ts_tablet_manager.cc:1939] T e1ebac187aa944d390d635b0a4673605 P 573aa84f695143b2a80922f87923b787: Deleting consensus metadata
I20260927 03:30:27.470091 13154 catalog_manager.cc:4841] TS 573aa84f695143b2a80922f87923b787 (127.0.98.3:41129): tablet e1ebac187aa944d390d635b0a4673605 (table test-table [id=f8d138b3c33d450bbbee691a13ca992a]) successfully deleted
I20260927 03:30:27.471727 13699 ts_tablet_manager.cc:1918] T e1ebac187aa944d390d635b0a4673605 P b6197d818093464395353fd3f75c1c6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:30:27.471817 13699 log.cc:1196] T e1ebac187aa944d390d635b0a4673605 P b6197d818093464395353fd3f75c1c6f: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790479675865103-392-0/minicluster-data/ts-1/wal/wals/e1ebac187aa944d390d635b0a4673605
I20260927 03:30:27.472110 13699 ts_tablet_manager.cc:1939] T e1ebac187aa944d390d635b0a4673605 P b6197d818093464395353fd3f75c1c6f: Deleting consensus metadata
I20260927 03:30:27.472566 13154 catalog_manager.cc:4841] TS b6197d818093464395353fd3f75c1c6f (127.0.98.2:42139): tablet e1ebac187aa944d390d635b0a4673605 (table test-table [id=f8d138b3c33d450bbbee691a13ca992a]) successfully deleted
W20260927 03:30:27.478317 13156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41631 (error 108)
I20260927 03:30:27.483999 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:27.483808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41631 (local address 127.0.0.1:59116)
0927 03:30:27.483873 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:27.483940 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 03:30:27.484239 13156 catalog_manager.cc:4584] TS 24a58ee4cf7e4326a8d0c129c0d68e9f (127.0.98.1:41631): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e1ebac187aa944d390d635b0a4673605: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41631: connect: Connection refused (error 111)
I20260927 03:30:27.491590 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 13372
I20260927 03:30:27.498517 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 13505
I20260927 03:30:27.506551 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 13137
I20260927 03:30:27.513803 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 03:30:27.858305 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 03:30:27.858398 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 03:30:28.202654 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 03:30:28.202 UTC [12204] LOG: received smart shutdown request
2026-09-27 03:30:28.206 UTC [12204] LOG: background worker "logical replication launcher" (PID 12213) exited with exit code 1
2026-09-27 03:30:28.206 UTC [12208] LOG: shutting down
2026-09-27 03:30:28.379 UTC [12204] LOG: database system is shut down
2026-09-27T03:30:28Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48376 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:30:28 dist-test-slave-j0hf krb5kdc[13710](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:30:28 dist-test-slave-j0hf krb5kdc[13710](info): set up 2 sockets
Sep 27 03:30:28 dist-test-slave-j0hf krb5kdc[13710](info): commencing operation
krb5kdc: starting...
W20260927 03:30:29.515301 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:30:29 dist-test-slave-j0hf krb5kdc[13710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:30:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:30:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:30:29.632076 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39417
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39417,127.0.98.61:36185,127.0.98.60:44163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:30:29.711369 13723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:30:29.711701 13723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:30:29.711738 13723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:30:29.712891 13723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:30:29.712955 13723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:30:29.712977 13723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:30:29.712998 13723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:30:29.713027 13723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:30:29.714555 13723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39417,127.0.98.61:36185,127.0.98.60:44163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:30:29.714823 13723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:30:29.715044 13723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:30:29.719776 13729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:29.719769 13728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:29.720036 13723 server_base.cc:1034] running on GCE node
W20260927 03:30:29.719770 13731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:29.720292 13723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:30:29.720587 13723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:30:29.721779 13723 hybrid_clock.cc:648] HybridClock initialized: now 1790479829721770 us; error 37 us; skew 500 ppm
Sep 27 03:30:29 dist-test-slave-j0hf krb5kdc[13710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:30:29.724402 13723 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:30:29.724848 13723 webserver.cc:460] Webserver started at http://127.0.98.62:33379/ using document root <none> and password file <none>
I20260927 03:30:29.725090 13723 fs_manager.cc:359] Metadata directory not provided
I20260927 03:30:29.725162 13723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:30:29.725313 13723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:30:29.726457 13723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "fe6fe71a618e41edb49a703a23ae0725"
format_stamp: "Formatted at 2026-09-27 03:30:29 on dist-test-slave-j0hf"
server_key: "7c2e6078d903f1915742a1465842709f"
server_key_iv: "7315ed332af5a425e92a56df76cee7ba"
server_key_version: "encryptionkey@0"
I20260927 03:30:29.726833 13723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "fe6fe71a618e41edb49a703a23ae0725"
format_stamp: "Formatted at 2026-09-27 03:30:29 on dist-test-slave-j0hf"
server_key: "7c2e6078d903f1915742a1465842709f"
server_key_iv: "7315ed332af5a425e92a56df76cee7ba"
server_key_version: "encryptionkey@0"
I20260927 03:30:29.728296 13723 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:30:29.729022 13738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:29.729215 13723 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:30:29.729319 13723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "fe6fe71a618e41edb49a703a23ae0725"
format_stamp: "Formatted at 2026-09-27 03:30:29 on dist-test-slave-j0hf"
server_key: "7c2e6078d903f1915742a1465842709f"
server_key_iv: "7315ed332af5a425e92a56df76cee7ba"
server_key_version: "encryptionkey@0"
I20260927 03:30:29.729398 13723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:30:29.747001 13723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:30:29.749377 13723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:30:29.749544 13723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:30:29.753405 13723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39417
I20260927 03:30:29.753439 13790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39417 every 8 connection(s)
I20260927 03:30:29.753808 13723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:30:29.754703 13791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:30:29.756318 13791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39417 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:30:29.756543 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 13723
I20260927 03:30:29.756660 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:30:29.756870 392 external_mini_cluster.cc:1428] Setting key 56044a52f329dbbb7d688b6c72685ab5
Sep 27 03:30:29 dist-test-slave-j0hf krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260927 03:30:29.765597 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:29.757059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60430 (local address 127.0.98.62:39417)
0927 03:30:29.757232 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:29.757240 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:29.757369 (+ 129us) server_negotiation.cc:407] Connection header received
0927 03:30:29.757898 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:29.757900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:29.757923 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:29.758012 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:29.758532 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:29.759329 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:29.760223 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:29.760533 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:29.760652 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:29.762581 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:29.762588 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:29.762588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:29.762594 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:29.764118 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:29.764467 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:29.764470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:29.764470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:29.764542 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:29.764747 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:29.764748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:29.764748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:29.764969 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:29.765065 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:29.765236 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 03:30:29.765353 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
I20260927 03:30:29.765601 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:29.757040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39417 (local address 127.0.0.1:60430)
0927 03:30:29.757242 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:29.757279 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:29.757737 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:29.758018 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:29.758020 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:29.758295 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:30:29.758435 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:29.758439 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:29.759465 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:29.759466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:29.760129 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:29.760130 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:29.760624 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:29.760625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:29.760685 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:29.761217 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:30:29.761224 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:30:29.762502 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:30:29.764202 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:29.764207 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:29.764209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:29.764384 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:29.764605 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:29.764606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:29.764606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:29.764688 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:29.765044 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:30:29.765048 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:30:29.765106 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 03:30:29.765200 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 03:30:29.765412 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":86,"thread_start_us":95,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260927 03:30:29.766384 13791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36185 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:30:29.766788 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:29.766653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36185 (local address 127.0.0.1:32814)
0927 03:30:29.766700 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:29.766762 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 03:30:29.766933 13740 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36185: connect: Connection refused (error 111)
I20260927 03:30:29.767313 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:29.767241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36185 (local address 127.0.0.1:32828)
0927 03:30:29.767264 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:29.767285 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260927 03:30:29.767432 13791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36185: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36185: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 03:30:29.780673 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36185
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39417,127.0.98.61:36185,127.0.98.60:44163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:30:29.816684 13791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36185 } attempt: 1
I20260927 03:30:29.817220 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:29.817086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36185 (local address 127.0.0.1:32834)
0927 03:30:29.817148 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:29.817190 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 03:30:29.817659 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:29.817563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36185 (local address 127.0.0.1:32850)
0927 03:30:29.817608 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:29.817632 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 03:30:29.817759 13791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36185: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36185: connect: Connection refused (error 111)
W20260927 03:30:29.862902 13798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:30:29.863142 13798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:30:29.863179 13798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:30:29.864301 13798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:30:29.864368 13798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:30:29.864398 13798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:30:29.864444 13798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:30:29.864470 13798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:30:29.865976 13798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39417,127.0.98.61:36185,127.0.98.60:44163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:30:29.866271 13798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:30:29.866493 13798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:30:29.871284 13804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:29.871384 13798 server_base.cc:1034] running on GCE node
W20260927 03:30:29.871249 13806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:29.871317 13803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:29.871721 13798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:30:29.871994 13798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:30:29.873194 13798 hybrid_clock.cc:648] HybridClock initialized: now 1790479829873181 us; error 27 us; skew 500 ppm
Sep 27 03:30:29 dist-test-slave-j0hf krb5kdc[13710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:30:29.875759 13798 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 03:30:29.876236 13798 webserver.cc:460] Webserver started at http://127.0.98.61:45497/ using document root <none> and password file <none>
I20260927 03:30:29.876444 13798 fs_manager.cc:359] Metadata directory not provided
I20260927 03:30:29.876514 13798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:30:29.876662 13798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:30:29.877697 13798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-1/data/instance:
uuid: "f6112cf9bfa1417bb7691bcabec37b68"
format_stamp: "Formatted at 2026-09-27 03:30:29 on dist-test-slave-j0hf"
server_key: "29e41f9ec57b9624c29221bc98f42d91"
server_key_iv: "86cea1e0fb3c5381c49febf5bd23a3e6"
server_key_version: "encryptionkey@0"
I20260927 03:30:29.878047 13798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-1/wal/instance:
uuid: "f6112cf9bfa1417bb7691bcabec37b68"
format_stamp: "Formatted at 2026-09-27 03:30:29 on dist-test-slave-j0hf"
server_key: "29e41f9ec57b9624c29221bc98f42d91"
server_key_iv: "86cea1e0fb3c5381c49febf5bd23a3e6"
server_key_version: "encryptionkey@0"
I20260927 03:30:29.879336 13798 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:30:29.880086 13813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:29.880249 13798 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:30:29.880343 13798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-1/wal
uuid: "f6112cf9bfa1417bb7691bcabec37b68"
format_stamp: "Formatted at 2026-09-27 03:30:29 on dist-test-slave-j0hf"
server_key: "29e41f9ec57b9624c29221bc98f42d91"
server_key_iv: "86cea1e0fb3c5381c49febf5bd23a3e6"
server_key_version: "encryptionkey@0"
I20260927 03:30:29.880421 13798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:30:29.885892 13791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36185 } attempt: 2
I20260927 03:30:29.886277 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:29.886141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36185 (local address 127.0.0.1:32856)
0927 03:30:29.886197 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:29.886239 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 03:30:29.886593 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:29.886517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36185 (local address 127.0.0.1:32870)
0927 03:30:29.886554 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:29.886577 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 03:30:29.886703 13791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36185: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36185: connect: Connection refused (error 111)
I20260927 03:30:29.892481 13798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:30:29.894670 13798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:30:29.894824 13798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:30:29.898898 13798 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36185
I20260927 03:30:29.898968 13865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36185 every 8 connection(s)
I20260927 03:30:29.899315 13798 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-1/data/info.pb
I20260927 03:30:29.904277 13866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:30:29.905328 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 13798
I20260927 03:30:29.905429 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-1/wal/instance
I20260927 03:30:29.905606 392 external_mini_cluster.cc:1428] Setting key 03ce35b4ef51bc0ee8b80b96b2de07bb
I20260927 03:30:29.907081 13866 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39417 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 03:30:29 dist-test-slave-j0hf krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:30:29.919411 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:29.907992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60432 (local address 127.0.98.62:39417)
0927 03:30:29.908067 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:29.908069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:29.909678 (+ 1609us) server_negotiation.cc:407] Connection header received
0927 03:30:29.910183 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:29.910184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:29.910197 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:29.910260 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:29.910877 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:29.911374 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:29.912393 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:29.912627 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:29.912710 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:29.916411 (+ 3701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:29.916412 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:29.916412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:29.916419 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:29.918065 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:29.918458 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:29.918459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:29.918460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:29.918514 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:29.918772 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:29.918773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:29.918773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:29.918885 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:29.918952 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:29.919238 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 03:30:29.919318 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:30:29.919643 13869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:29.909361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39417 (local address 127.0.0.1:60432)
0927 03:30:29.909538 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:29.909581 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:29.910078 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:29.910331 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:29.910335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:29.910598 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:30:29.910786 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:29.910790 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:29.911501 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:29.911502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:29.912302 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:29.912304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:29.912719 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:29.912720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:29.912844 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:29.913400 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:30:29.913408 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:30:29.916319 (+ 2911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:30:29.918150 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:29.918157 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:29.918159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:29.918387 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:29.918596 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:29.918597 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:29.918597 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:29.918703 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:29.918944 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:30:29.918950 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:30:29.919111 (+ 161us) client_negotiation.cc:769] Sending connection context
0927 03:30:29.919218 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 03:30:29.919373 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20260927 03:30:29.920207 13866 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36185 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:30:29 dist-test-slave-j0hf krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 03:30:29.927727 13869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:29.920446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36185 (local address 127.0.0.1:32876)
0927 03:30:29.920488 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:29.920499 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:29.920537 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:29.920760 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:29.920761 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:29.920908 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:30:29.920950 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:29.920951 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:29.921797 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:29.921798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:29.922309 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:29.922310 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:29.922775 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:29.922776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:29.922838 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:29.923187 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:30:29.923190 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:30:29.924815 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:30:29.926674 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:29.926675 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:29.926675 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:29.926875 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:29.927101 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:29.927101 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:29.927102 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:29.927155 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:29.927505 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:30:29.927505 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:30:29.927540 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 03:30:29.927597 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 03:30:29.927668 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 03:30:29.927889 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:29.920492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32876 (local address 127.0.98.61:36185)
0927 03:30:29.920632 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:29.920636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:29.920651 (+ 15us) server_negotiation.cc:407] Connection header received
0927 03:30:29.920669 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:29.920671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:29.920687 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:29.920759 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:29.921024 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:29.921709 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:29.922396 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:29.922691 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:29.922775 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:29.924892 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:29.924899 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:29.924899 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:29.924906 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:29.926594 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:29.926942 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:29.926945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:29.926945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:29.927004 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:29.927209 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:29.927209 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:29.927210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:29.927409 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:29.927520 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:29.927620 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 03:30:29.927723 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
I20260927 03:30:29.928455 13866 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44163 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:30:29.928794 13869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:29.928681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44163 (local address 127.0.0.1:46050)
0927 03:30:29.928725 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:29.928769 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 03:30:29.928908 13815 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44163: connect: Connection refused (error 111)
I20260927 03:30:29.929340 13869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:29.929257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44163 (local address 127.0.0.1:46062)
0927 03:30:29.929297 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:29.929320 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 03:30:29.929437 13866 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44163: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44163: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 03:30:29.930258 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44163
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39417,127.0.98.61:36185,127.0.98.60:44163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:30:29.977865 13791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36185 } attempt: 3
I20260927 03:30:29.978639 13866 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44163 } attempt: 1
I20260927 03:30:29.979279 13869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:29.979137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44163 (local address 127.0.0.1:46078)
0927 03:30:29.979197 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:29.979242 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 03:30:29.979708 13869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:29.979599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44163 (local address 127.0.0.1:46080)
0927 03:30:29.979650 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:29.979683 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 03:30:29.979846 13866 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44163: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44163: connect: Connection refused (error 111)
Sep 27 03:30:29 dist-test-slave-j0hf krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 03:30:29.985311 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:29.978201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36185 (local address 127.0.0.1:32892)
0927 03:30:29.978282 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:29.978297 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:29.978370 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:29.978696 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:29.978698 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:29.978904 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:30:29.978969 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:29.978971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:29.979919 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:29.979920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:29.980460 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:29.980463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:29.980854 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:29.980854 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:29.980931 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:29.981422 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:30:29.981427 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:30:29.982559 (+ 1132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:30:29.984308 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:29.984309 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:29.984309 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:29.984475 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:29.984680 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:29.984681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:29.984681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:29.984738 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:29.984984 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:30:29.984984 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:30:29.985032 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 03:30:29.985118 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 03:30:29.985218 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 03:30:29.985307 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:29.978474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32892 (local address 127.0.98.61:36185)
0927 03:30:29.978540 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:29.978543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:29.978553 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:30:29.978595 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:29.978596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:29.978608 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:29.978672 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:29.979042 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:29.979810 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:29.980545 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:29.980769 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:29.980852 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:29.982640 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:29.982641 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:29.982642 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:29.982649 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:29.984223 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:29.984560 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:29.984561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:29.984562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:29.984618 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:29.984804 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:29.984805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:29.984806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:29.984920 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:29.984992 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:29.985141 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 03:30:29.985218 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 03:30:29.985656 13791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44163 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:30:29.985972 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:29.985881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44163 (local address 127.0.0.1:46094)
0927 03:30:29.985924 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:29.985948 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 03:30:29.986284 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:29.986197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44163 (local address 127.0.0.1:46110)
0927 03:30:29.986241 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:29.986264 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 03:30:29.986395 13791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44163: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44163: connect: Connection refused (error 111)
W20260927 03:30:30.015008 13873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:30:30.015247 13873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:30:30.015272 13873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:30:30.016495 13873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:30:30.016541 13873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:30:30.016557 13873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:30:30.016613 13873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:30:30.016642 13873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:30:30.017529 13791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44163 } attempt: 1
I20260927 03:30:30.017926 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:30.017801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44163 (local address 127.0.0.1:46116)
0927 03:30:30.017861 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:30.017899 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 03:30:30.018332 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:30.018246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44163 (local address 127.0.0.1:46118)
0927 03:30:30.018291 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:30.018316 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 03:30:30.018326 13873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39417,127.0.98.61:36185,127.0.98.60:44163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
W20260927 03:30:30.018455 13791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44163: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44163: connect: Connection refused (error 111)
I20260927 03:30:30.018571 13873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:30:30.018757 13873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:30:30.023494 13881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:30.023518 13873 server_base.cc:1034] running on GCE node
W20260927 03:30:30.023500 13878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:30.023507 13879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:30.023931 13873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:30:30.024195 13873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:30:30.025357 13873 hybrid_clock.cc:648] HybridClock initialized: now 1790479830025341 us; error 30 us; skew 500 ppm
Sep 27 03:30:30 dist-test-slave-j0hf krb5kdc[13710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:30:30.027681 13873 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 03:30:30.028079 13873 webserver.cc:460] Webserver started at http://127.0.98.60:37215/ using document root <none> and password file <none>
I20260927 03:30:30.028286 13873 fs_manager.cc:359] Metadata directory not provided
I20260927 03:30:30.028359 13873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:30:30.028494 13873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:30:30.029492 13873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-2/data/instance:
uuid: "7a7ce359178340d28567d556dc6d0290"
format_stamp: "Formatted at 2026-09-27 03:30:30 on dist-test-slave-j0hf"
server_key: "6445563c3c5014379ef520374168c872"
server_key_iv: "de476d1ba5716a481eb8d224e0a274a4"
server_key_version: "encryptionkey@0"
I20260927 03:30:30.029865 13873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-2/wal/instance:
uuid: "7a7ce359178340d28567d556dc6d0290"
format_stamp: "Formatted at 2026-09-27 03:30:30 on dist-test-slave-j0hf"
server_key: "6445563c3c5014379ef520374168c872"
server_key_iv: "de476d1ba5716a481eb8d224e0a274a4"
server_key_version: "encryptionkey@0"
I20260927 03:30:30.031119 13873 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:30:30.031880 13888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:30.032071 13873 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:30.032146 13873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-2/wal
uuid: "7a7ce359178340d28567d556dc6d0290"
format_stamp: "Formatted at 2026-09-27 03:30:30 on dist-test-slave-j0hf"
server_key: "6445563c3c5014379ef520374168c872"
server_key_iv: "de476d1ba5716a481eb8d224e0a274a4"
server_key_version: "encryptionkey@0"
I20260927 03:30:30.032218 13873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:30:30.039882 13873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:30:30.041666 13873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:30:30.041800 13873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:30:30.045624 13873 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44163
I20260927 03:30:30.045652 13940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44163 every 8 connection(s)
I20260927 03:30:30.045979 13873 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-2/data/info.pb
I20260927 03:30:30.046845 13941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:30:30.048000 13866 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44163 } attempt: 2
I20260927 03:30:30.048725 13941 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39417 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:30:30 dist-test-slave-j0hf krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 03:30:30 dist-test-slave-j0hf krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:30:30.055058 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 13873
I20260927 03:30:30.055204 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790479675865103-392-0/minicluster-data/master-2/wal/instance
I20260927 03:30:30.055454 392 external_mini_cluster.cc:1428] Setting key 4e6f7c16167a3e1db4df0a1d6b42e258
I20260927 03:30:30.057699 13869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:30.048273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44163 (local address 127.0.0.1:46134)
0927 03:30:30.048325 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:30.048337 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:30.048386 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:30.049183 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:30.049185 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:30.049364 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:30:30.049414 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:30.049417 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:30.050569 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:30.050570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:30.051065 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:30.051067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:30.051710 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:30.051712 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:30.051767 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:30.052196 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:30:30.052202 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:30:30.053724 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:30:30.056220 (+ 2496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:30.056221 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:30.056221 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:30.056470 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:30.056771 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:30.056772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:30.056773 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:30.056850 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:30.057328 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:30:30.057329 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:30:30.057380 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 03:30:30.057475 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 03:30:30.057578 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 03:30:30.057896 13943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:30.048414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46134 (local address 127.0.98.60:44163)
0927 03:30:30.048573 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:30.048581 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:30.048610 (+ 29us) server_negotiation.cc:407] Connection header received
0927 03:30:30.048996 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:30.049002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:30.049045 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:30.049159 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:30.049522 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:30.050477 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:30.051157 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:30.051619 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:30.051741 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:30.053819 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:30.053826 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:30.053827 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:30.053833 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:30.056104 (+ 2271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:30.056579 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:30.056582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:30.056583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:30.056689 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:30.056923 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:30.056924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:30.056925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:30.057226 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:30.057343 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:30.057551 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 03:30:30.057686 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
I20260927 03:30:30.058089 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:30.049644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39417 (local address 127.0.0.1:60444)
0927 03:30:30.049804 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:30.049835 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:30.049907 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:30.050095 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:30.050098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:30.050340 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:30:30.050426 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:30.050430 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:30.051055 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:30.051056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:30.051699 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:30.051702 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:30.052071 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:30.052072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:30.052142 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:30.052576 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:30:30.052584 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:30:30.054322 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:30:30.056817 (+ 2495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:30.056822 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:30.056824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:30.057092 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:30.057312 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:30.057313 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:30.057313 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:30.057370 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:30.057676 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:30:30.057682 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:30:30.057733 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 03:30:30.057828 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 03:30:30.057937 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":88,"threads_started":1}
I20260927 03:30:30.058444 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:30.049696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60444 (local address 127.0.98.62:39417)
0927 03:30:30.049751 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:30.049753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:30.049889 (+ 136us) server_negotiation.cc:407] Connection header received
0927 03:30:30.049969 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:30.049970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:30.050004 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:30.050061 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:30.050519 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:30.050967 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:30.051786 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:30.051966 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:30.052064 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:30.054406 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:30.054408 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:30.054408 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:30.054416 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:30.056731 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:30.057176 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:30.057177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:30.057177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:30.057237 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:30.057426 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:30.057427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:30.057428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:30.057577 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:30.057644 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:30.058217 (+ 573us) server_negotiation.cc:299] Negotiation successful
0927 03:30:30.058323 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 03:30:30.058804 13941 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36185 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:30:30.059923 13866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68: Bootstrap starting.
I20260927 03:30:30.060971 13866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68: Neither blocks nor log segments found. Creating new log.
Sep 27 03:30:30 dist-test-slave-j0hf krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 03:30:30.061295 13866 log.cc:824] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68: Log is configured to *not* fsync() on all Append() calls
I20260927 03:30:30.062091 13866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68: No bootstrap required, opened a new log
Sep 27 03:30:30 dist-test-slave-j0hf krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 03:30:30.064610 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:30.055818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46148 (local address 127.0.98.60:44163)
0927 03:30:30.055964 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:30.055974 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:30.056065 (+ 91us) server_negotiation.cc:407] Connection header received
0927 03:30:30.056143 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:30.056144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:30.056154 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:30.056213 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:30.056817 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:30.057430 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:30.058744 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:30.059215 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:30.059284 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:30.061845 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:30.061846 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:30.061847 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:30.061853 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:30.063506 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:30.063805 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:30.063806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:30.063807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:30.063859 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:30.064051 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:30.064052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:30.064052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:30.064155 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:30.064290 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:30.064369 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 03:30:30.064484 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20260927 03:30:30.064553 13866 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6fe71a618e41edb49a703a23ae0725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39417 } } peers { permanent_uuid: "f6112cf9bfa1417bb7691bcabec37b68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36185 } } peers { permanent_uuid: "7a7ce359178340d28567d556dc6d0290" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44163 } }
I20260927 03:30:30.064693 13866 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:30:30.064733 13866 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6112cf9bfa1417bb7691bcabec37b68, State: Initialized, Role: FOLLOWER
I20260927 03:30:30.064872 13866 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6fe71a618e41edb49a703a23ae0725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39417 } } peers { permanent_uuid: "f6112cf9bfa1417bb7691bcabec37b68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36185 } } peers { permanent_uuid: "7a7ce359178340d28567d556dc6d0290" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44163 } }
I20260927 03:30:30.065171 13948 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6fe71a618e41edb49a703a23ae0725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39417 } } peers { permanent_uuid: "f6112cf9bfa1417bb7691bcabec37b68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36185 } } peers { permanent_uuid: "7a7ce359178340d28567d556dc6d0290" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44163 } } }
I20260927 03:30:30.065224 13866 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:30:30.065256 13948 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:30:30.067745 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:30.059114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32904 (local address 127.0.98.61:36185)
0927 03:30:30.059172 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:30.059174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:30.059212 (+ 38us) server_negotiation.cc:407] Connection header received
0927 03:30:30.059303 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:30.059304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:30.059314 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:30.059368 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:30.059986 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:30.060675 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:30.061718 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:30.062036 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:30.062110 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:30.064861 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:30.064862 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:30.064862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:30.064869 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:30.066444 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:30.066879 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:30.066880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:30.066881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:30.066956 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:30.067185 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:30.067186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:30.067186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:30.067336 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:30.067378 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:30.067595 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 03:30:30.067664 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 03:30:30.068084 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:30.059047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36185 (local address 127.0.0.1:32904)
0927 03:30:30.059160 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:30.059171 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:30.059226 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:30.059372 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:30.059373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:30.059835 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:30:30.059889 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:30.059891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:30.060782 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:30.060783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:30.061630 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:30.061633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:30.062386 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:30.062387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:30.062472 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:30.063073 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:30:30.063078 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:30:30.064737 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:30:30.066533 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:30.066534 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:30.066535 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:30.066757 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:30.067039 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:30.067040 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:30.067040 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:30.067105 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:30.067449 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:30:30.067449 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:30:30.067488 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 03:30:30.067893 (+ 405us) client_negotiation.cc:240] Negotiation successful
0927 03:30:30.067998 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260927 03:30:30.068881 13941 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44163 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:30:30.068013 13798 master_runner.cc:186] Error getting master registration for 127.0.98.62:39417: OK, instance_id { permanent_uuid: "fe6fe71a618e41edb49a703a23ae0725" instance_seqno: 1790479829752282 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
Sep 27 03:30:30 dist-test-slave-j0hf krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:30:30.071185 13798 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 27 03:30:30 dist-test-slave-j0hf krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
W20260927 03:30:30.073167 13959 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:30:30.073221 13959 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:30:30.075878 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:30.066280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60460 (local address 127.0.98.62:39417)
0927 03:30:30.066334 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:30.066336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:30.066345 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:30:30.066369 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:30.066370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:30.066381 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:30.066507 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:30.067140 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:30.067869 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:30.068847 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:30.069130 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:30.069186 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:30.072665 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:30.072666 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:30.072666 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:30.072673 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:30.074461 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:30.074885 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:30.074886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:30.074886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:30.074944 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:30.075172 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:30.075173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:30.075173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:30.075304 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:30.075528 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:30.075703 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 03:30:30.075778 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 03:30:30.076437 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:30.069078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44163 (local address 127.0.0.1:46158)
0927 03:30:30.069274 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:30.069283 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:30.069318 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:30.069458 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:30.069459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:30.069591 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:30:30.069635 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:30.069636 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:30.070230 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:30.070231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:30.071042 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:30.071044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:30.071527 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:30.071528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:30.071575 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:30.071944 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:30:30.071947 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:30:30.073125 (+ 1178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:30:30.075055 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:30.075056 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:30.075056 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:30.075259 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:30.075539 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:30.075540 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:30.075541 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:30.075606 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:30.076171 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:30:30.076171 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:30:30.076208 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 03:30:30.076255 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 03:30:30.076340 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189}
I20260927 03:30:30.076762 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:30.069225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46158 (local address 127.0.98.60:44163)
0927 03:30:30.069256 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:30.069258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:30.069363 (+ 105us) server_negotiation.cc:407] Connection header received
0927 03:30:30.069378 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:30.069379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:30.069387 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:30.069427 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:30.069698 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:30.070155 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:30.071133 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:30.071408 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:30.071530 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:30.073221 (+ 1691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:30.073222 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:30.073222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:30.073228 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:30.074999 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:30.075335 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:30.075336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:30.075337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:30.075400 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:30.075911 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:30.075912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:30.075912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:30.076084 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:30.076563 (+ 479us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:30.076606 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 03:30:30.076684 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 03:30:30.078220 13941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a7ce359178340d28567d556dc6d0290: Bootstrap starting.
I20260927 03:30:30.078943 13941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a7ce359178340d28567d556dc6d0290: Neither blocks nor log segments found. Creating new log.
I20260927 03:30:30.079245 13941 log.cc:824] T 00000000000000000000000000000000 P 7a7ce359178340d28567d556dc6d0290: Log is configured to *not* fsync() on all Append() calls
I20260927 03:30:30.080420 13941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a7ce359178340d28567d556dc6d0290: No bootstrap required, opened a new log
Sep 27 03:30:30 dist-test-slave-j0hf krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 03:30:30.082711 13941 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a7ce359178340d28567d556dc6d0290 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6fe71a618e41edb49a703a23ae0725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39417 } } peers { permanent_uuid: "f6112cf9bfa1417bb7691bcabec37b68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36185 } } peers { permanent_uuid: "7a7ce359178340d28567d556dc6d0290" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44163 } }
I20260927 03:30:30.082873 13941 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a7ce359178340d28567d556dc6d0290 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:30:30.082922 13941 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a7ce359178340d28567d556dc6d0290 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a7ce359178340d28567d556dc6d0290, State: Initialized, Role: FOLLOWER
I20260927 03:30:30.083050 13941 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a7ce359178340d28567d556dc6d0290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6fe71a618e41edb49a703a23ae0725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39417 } } peers { permanent_uuid: "f6112cf9bfa1417bb7691bcabec37b68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36185 } } peers { permanent_uuid: "7a7ce359178340d28567d556dc6d0290" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44163 } }
I20260927 03:30:30.083386 13941 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a7ce359178340d28567d556dc6d0290 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:30:30.084954 13962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a7ce359178340d28567d556dc6d0290 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6fe71a618e41edb49a703a23ae0725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39417 } } peers { permanent_uuid: "f6112cf9bfa1417bb7691bcabec37b68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36185 } } peers { permanent_uuid: "7a7ce359178340d28567d556dc6d0290" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44163 } } }
I20260927 03:30:30.085196 13962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a7ce359178340d28567d556dc6d0290 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:30:30.085654 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:30.076418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32914 (local address 127.0.98.61:36185)
0927 03:30:30.076452 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:30.076454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:30.076729 (+ 275us) server_negotiation.cc:407] Connection header received
0927 03:30:30.076747 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:30.076748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:30.076756 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:30.076818 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:30.077553 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:30.078220 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:30.079139 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:30.079417 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:30.079493 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:30.082158 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:30.082160 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:30.082160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:30.082166 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:30.084000 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:30.084442 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:30.084443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:30.084443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:30.084503 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:30.084737 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:30.084738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:30.084739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:30.084879 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:30.085037 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:30.085487 (+ 450us) server_negotiation.cc:299] Negotiation successful
0927 03:30:30.085565 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
W20260927 03:30:30.085954 13973 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a7ce359178340d28567d556dc6d0290: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:30:30.086047 13973 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:30:30.087306 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:30:30.085501 13873 master_runner.cc:186] Error getting master registration for 127.0.98.62:39417: OK, instance_id { permanent_uuid: "fe6fe71a618e41edb49a703a23ae0725" instance_seqno: 1790479829752282 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 03:30:30.087889 13873 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 03:30:30.093583 13791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44163 } attempt: 2
I20260927 03:30:30.096782 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:30.088755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60468 (local address 127.0.98.62:39417)
0927 03:30:30.088802 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:30.088804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:30.088976 (+ 172us) server_negotiation.cc:407] Connection header received
0927 03:30:30.089131 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:30.089132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:30.089143 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:30.089178 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:30.090007 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:30.090672 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:30.091368 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:30.091681 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:30.091746 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:30.093276 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:30.093278 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:30.093278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:30.093285 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:30.095320 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:30.095747 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:30.095748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:30.095748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:30.095805 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:30.096013 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:30.096014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:30.096015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:30.096139 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:30.096241 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:30.096613 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 03:30:30.096689 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Sep 27 03:30:30 dist-test-slave-j0hf krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 03:30:30.104497 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:30.093815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44163 (local address 127.0.0.1:46180)
0927 03:30:30.093860 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:30.093873 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:30.093919 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:30.094059 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:30.094060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:30.094232 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:30:30.094272 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:30.094273 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:30.095494 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:30.095495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:30.096245 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:30.096248 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:30.096785 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:30.096786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:30.096841 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:30.097534 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:30:30.097539 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:30:30.099025 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:30:30.103403 (+ 4378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:30.103404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:30.103405 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:30.103649 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:30.103855 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:30.103856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:30.103856 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:30.103910 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:30.104187 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:30:30.104188 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:30:30.104232 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 03:30:30.104294 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 03:30:30.104396 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 03:30:30.104861 13943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:30.093909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46180 (local address 127.0.98.60:44163)
0927 03:30:30.093943 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:30.093945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:30.093954 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:30:30.093976 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:30.093977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:30.093996 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:30.094055 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:30.094368 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:30.095357 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:30.096349 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:30.096666 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:30.096729 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:30.099128 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:30.099129 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:30.099129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:30.099137 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:30.102629 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:30.103732 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:30.103733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:30.103733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:30.103787 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:30.103972 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:30.103973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:30.103973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:30.104103 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:30.104161 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:30.104693 (+ 532us) server_negotiation.cc:299] Negotiation successful
0927 03:30:30.104772 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:30:30.106046 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:30.088944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32920 (local address 127.0.98.61:36185)
0927 03:30:30.089010 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:30.089011 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:30.089045 (+ 34us) server_negotiation.cc:407] Connection header received
0927 03:30:30.089208 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:30.089209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:30.089219 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:30.089274 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:30.089805 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:30.090517 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:30.091566 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:30.091851 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:30.091894 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:30.102377 (+ 10483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:30.102379 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:30.102379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:30.102386 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:30.104134 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:30.104856 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:30.104857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:30.104858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:30.105197 (+ 339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:30.105436 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:30.105437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:30.105438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:30.105582 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:30.105654 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:30.105867 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 03:30:30.105952 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 03:30:30.106371 13791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe6fe71a618e41edb49a703a23ae0725: Bootstrap starting.
I20260927 03:30:30.107110 13791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe6fe71a618e41edb49a703a23ae0725: Neither blocks nor log segments found. Creating new log.
I20260927 03:30:30.107441 13791 log.cc:824] T 00000000000000000000000000000000 P fe6fe71a618e41edb49a703a23ae0725: Log is configured to *not* fsync() on all Append() calls
I20260927 03:30:30.108727 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:30.089072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46168 (local address 127.0.98.60:44163)
0927 03:30:30.089110 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:30.089112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:30.097226 (+ 8114us) server_negotiation.cc:407] Connection header received
0927 03:30:30.097318 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:30.097319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:30.097328 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:30.097357 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:30.097882 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:30.098582 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:30.099582 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:30.099879 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:30.099918 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:30.101302 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:30.101303 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:30.101303 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:30.101310 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:30.103314 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:30.107896 (+ 4582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:30.107897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:30.107897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:30.107954 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:30.108190 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:30.108191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:30.108191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:30.108312 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:30.108414 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:30.108564 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 03:30:30.108638 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:30:30.112151 13791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe6fe71a618e41edb49a703a23ae0725: No bootstrap required, opened a new log
I20260927 03:30:30.114625 13791 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe6fe71a618e41edb49a703a23ae0725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6fe71a618e41edb49a703a23ae0725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39417 } } peers { permanent_uuid: "f6112cf9bfa1417bb7691bcabec37b68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36185 } } peers { permanent_uuid: "7a7ce359178340d28567d556dc6d0290" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44163 } }
I20260927 03:30:30.114768 13791 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe6fe71a618e41edb49a703a23ae0725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:30:30.114802 13791 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe6fe71a618e41edb49a703a23ae0725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe6fe71a618e41edb49a703a23ae0725, State: Initialized, Role: FOLLOWER
I20260927 03:30:30.114926 13791 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe6fe71a618e41edb49a703a23ae0725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6fe71a618e41edb49a703a23ae0725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39417 } } peers { permanent_uuid: "f6112cf9bfa1417bb7691bcabec37b68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36185 } } peers { permanent_uuid: "7a7ce359178340d28567d556dc6d0290" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44163 } }
I20260927 03:30:30.115262 13791 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe6fe71a618e41edb49a703a23ae0725 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:30:30.115927 13983 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe6fe71a618e41edb49a703a23ae0725 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6fe71a618e41edb49a703a23ae0725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39417 } } peers { permanent_uuid: "f6112cf9bfa1417bb7691bcabec37b68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36185 } } peers { permanent_uuid: "7a7ce359178340d28567d556dc6d0290" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44163 } } }
I20260927 03:30:30.116086 13983 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe6fe71a618e41edb49a703a23ae0725 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 03:30:30.117462 13994 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fe6fe71a618e41edb49a703a23ae0725: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:30:30.117570 13994 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:30:30.119953 13723 master_runner.cc:263] No leader master found from master fe6fe71a618e41edb49a703a23ae0725
I20260927 03:30:30.120015 13723 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 03:30:30.145467 13819 tablet.cc:2369] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:30:30.250350 13744 tablet.cc:2369] T 00000000000000000000000000000000 P fe6fe71a618e41edb49a703a23ae0725: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:30:30.250701 13948 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:30:30.250814 13948 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6fe71a618e41edb49a703a23ae0725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39417 } } peers { permanent_uuid: "f6112cf9bfa1417bb7691bcabec37b68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36185 } } peers { permanent_uuid: "7a7ce359178340d28567d556dc6d0290" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44163 } }
I20260927 03:30:30.251073 13948 leader_election.cc:288] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe6fe71a618e41edb49a703a23ae0725 (127.0.98.62:39417), 7a7ce359178340d28567d556dc6d0290 (127.0.98.60:44163)
I20260927 03:30:30.251366 13915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f6112cf9bfa1417bb7691bcabec37b68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7ce359178340d28567d556dc6d0290" is_pre_election: true
I20260927 03:30:30.251405 13765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f6112cf9bfa1417bb7691bcabec37b68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe6fe71a618e41edb49a703a23ae0725" is_pre_election: true
I20260927 03:30:30.251560 13915 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a7ce359178340d28567d556dc6d0290 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6112cf9bfa1417bb7691bcabec37b68 in term 0.
I20260927 03:30:30.251621 13765 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fe6fe71a618e41edb49a703a23ae0725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6112cf9bfa1417bb7691bcabec37b68 in term 0.
I20260927 03:30:30.251755 13815 leader_election.cc:302] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a7ce359178340d28567d556dc6d0290, f6112cf9bfa1417bb7691bcabec37b68; no voters:
I20260927 03:30:30.251940 13948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:30:30.251996 13948 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:30:30.252027 13948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:30:30.252892 13948 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6fe71a618e41edb49a703a23ae0725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39417 } } peers { permanent_uuid: "f6112cf9bfa1417bb7691bcabec37b68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36185 } } peers { permanent_uuid: "7a7ce359178340d28567d556dc6d0290" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44163 } }
I20260927 03:30:30.253022 13948 leader_election.cc:288] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68 [CANDIDATE]: Term 1 election: Requested vote from peers fe6fe71a618e41edb49a703a23ae0725 (127.0.98.62:39417), 7a7ce359178340d28567d556dc6d0290 (127.0.98.60:44163)
I20260927 03:30:30.253224 13765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f6112cf9bfa1417bb7691bcabec37b68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe6fe71a618e41edb49a703a23ae0725"
I20260927 03:30:30.253237 13915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f6112cf9bfa1417bb7691bcabec37b68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7ce359178340d28567d556dc6d0290"
I20260927 03:30:30.253295 13765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe6fe71a618e41edb49a703a23ae0725 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:30:30.253317 13915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a7ce359178340d28567d556dc6d0290 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:30:30.254213 13765 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fe6fe71a618e41edb49a703a23ae0725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6112cf9bfa1417bb7691bcabec37b68 in term 1.
I20260927 03:30:30.254215 13915 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a7ce359178340d28567d556dc6d0290 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6112cf9bfa1417bb7691bcabec37b68 in term 1.
I20260927 03:30:30.254366 13815 leader_election.cc:302] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f6112cf9bfa1417bb7691bcabec37b68, fe6fe71a618e41edb49a703a23ae0725; no voters:
I20260927 03:30:30.254470 13948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:30:30.254629 13948 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68 [term 1 LEADER]: Becoming Leader. State: Replica: f6112cf9bfa1417bb7691bcabec37b68, State: Running, Role: LEADER
I20260927 03:30:30.254716 13948 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6fe71a618e41edb49a703a23ae0725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39417 } } peers { permanent_uuid: "f6112cf9bfa1417bb7691bcabec37b68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36185 } } peers { permanent_uuid: "7a7ce359178340d28567d556dc6d0290" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44163 } }
I20260927 03:30:30.255103 13996 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6112cf9bfa1417bb7691bcabec37b68. Latest consensus state: current_term: 1 leader_uuid: "f6112cf9bfa1417bb7691bcabec37b68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6fe71a618e41edb49a703a23ae0725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39417 } } peers { permanent_uuid: "f6112cf9bfa1417bb7691bcabec37b68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36185 } } peers { permanent_uuid: "7a7ce359178340d28567d556dc6d0290" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44163 } } }
I20260927 03:30:30.255244 13996 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68 [sys.catalog]: This master's current role is: LEADER
I20260927 03:30:30.255527 13998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:30:30.255749 13998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:30:30.257148 13765 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fe6fe71a618e41edb49a703a23ae0725 [term 1 FOLLOWER]: Refusing update from remote peer f6112cf9bfa1417bb7691bcabec37b68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:30:30.257177 13915 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7a7ce359178340d28567d556dc6d0290 [term 1 FOLLOWER]: Refusing update from remote peer f6112cf9bfa1417bb7691bcabec37b68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:30:30.257472 13996 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a7ce359178340d28567d556dc6d0290" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:30:30.257581 13948 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe6fe71a618e41edb49a703a23ae0725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:30:30.258765 13983 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe6fe71a618e41edb49a703a23ae0725 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6112cf9bfa1417bb7691bcabec37b68. Latest consensus state: current_term: 1 leader_uuid: "f6112cf9bfa1417bb7691bcabec37b68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6fe71a618e41edb49a703a23ae0725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39417 } } peers { permanent_uuid: "f6112cf9bfa1417bb7691bcabec37b68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36185 } } peers { permanent_uuid: "7a7ce359178340d28567d556dc6d0290" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44163 } } }
I20260927 03:30:30.258852 13983 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe6fe71a618e41edb49a703a23ae0725 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:30:30.258836 13962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a7ce359178340d28567d556dc6d0290 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6112cf9bfa1417bb7691bcabec37b68. Latest consensus state: current_term: 1 leader_uuid: "f6112cf9bfa1417bb7691bcabec37b68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6fe71a618e41edb49a703a23ae0725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39417 } } peers { permanent_uuid: "f6112cf9bfa1417bb7691bcabec37b68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36185 } } peers { permanent_uuid: "7a7ce359178340d28567d556dc6d0290" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44163 } } }
I20260927 03:30:30.258945 13962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a7ce359178340d28567d556dc6d0290 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:30:30.259974 13948 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f6112cf9bfa1417bb7691bcabec37b68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6fe71a618e41edb49a703a23ae0725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39417 } } peers { permanent_uuid: "f6112cf9bfa1417bb7691bcabec37b68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36185 } } peers { permanent_uuid: "7a7ce359178340d28567d556dc6d0290" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44163 } } }
I20260927 03:30:30.260053 13948 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68 [sys.catalog]: This master's current role is: LEADER
I20260927 03:30:30.260092 13996 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f6112cf9bfa1417bb7691bcabec37b68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6fe71a618e41edb49a703a23ae0725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39417 } } peers { permanent_uuid: "f6112cf9bfa1417bb7691bcabec37b68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36185 } } peers { permanent_uuid: "7a7ce359178340d28567d556dc6d0290" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44163 } } }
I20260927 03:30:30.260164 13996 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68 [sys.catalog]: This master's current role is: LEADER
I20260927 03:30:30.260538 13962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a7ce359178340d28567d556dc6d0290 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f6112cf9bfa1417bb7691bcabec37b68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6fe71a618e41edb49a703a23ae0725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39417 } } peers { permanent_uuid: "f6112cf9bfa1417bb7691bcabec37b68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36185 } } peers { permanent_uuid: "7a7ce359178340d28567d556dc6d0290" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44163 } } }
I20260927 03:30:30.260612 13962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a7ce359178340d28567d556dc6d0290 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:30:30.260602 13999 mvcc.cc:205] Tried to move back new op lower bound from 7333805384731250688 to 7333805384723832832. Current Snapshot: MvccSnapshot[applied={T|T < 7333805384731250688}]
I20260927 03:30:30.260890 14001 mvcc.cc:205] Tried to move back new op lower bound from 7333805384731250688 to 7333805384723832832. Current Snapshot: MvccSnapshot[applied={T|T < 7333805384731250688}]
I20260927 03:30:30.261148 13983 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe6fe71a618e41edb49a703a23ae0725 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f6112cf9bfa1417bb7691bcabec37b68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6fe71a618e41edb49a703a23ae0725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39417 } } peers { permanent_uuid: "f6112cf9bfa1417bb7691bcabec37b68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36185 } } peers { permanent_uuid: "7a7ce359178340d28567d556dc6d0290" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44163 } } }
I20260927 03:30:30.261220 13983 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe6fe71a618e41edb49a703a23ae0725 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:30:30.261356 13998 catalog_manager.cc:1347] Generated new cluster ID: f68b0ee8e1b044beb1a4c02f7511e3f4
I20260927 03:30:30.261392 13998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:30:30.269878 13998 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:30:30.270601 13998 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:30:30.275199 13998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6112cf9bfa1417bb7691bcabec37b68: Generated new TSK 0
I20260927 03:30:30.275367 13998 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:30:31.086522 13973 catalog_manager.cc:1259] Loaded cluster ID: f68b0ee8e1b044beb1a4c02f7511e3f4
I20260927 03:30:31.086604 13973 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7a7ce359178340d28567d556dc6d0290: loading cluster ID for follower catalog manager: success
I20260927 03:30:31.087504 13973 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7a7ce359178340d28567d556dc6d0290: acquiring CA information for follower catalog manager: success
I20260927 03:30:31.087810 13973 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7a7ce359178340d28567d556dc6d0290: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 03:30:31.118017 13994 catalog_manager.cc:1259] Loaded cluster ID: f68b0ee8e1b044beb1a4c02f7511e3f4
I20260927 03:30:31.118103 13994 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fe6fe71a618e41edb49a703a23ae0725: loading cluster ID for follower catalog manager: success
I20260927 03:30:31.118968 13994 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fe6fe71a618e41edb49a703a23ae0725: acquiring CA information for follower catalog manager: success
I20260927 03:30:31.119294 13994 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fe6fe71a618e41edb49a703a23ae0725: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 03:30:31.323840 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 13723
I20260927 03:30:31.329178 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 13798
I20260927 03:30:31.334113 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 13873
2026-09-27T03:30:31Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2846 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:30:31 dist-test-slave-j0hf krb5kdc[14014](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:30:31 dist-test-slave-j0hf krb5kdc[14014](info): set up 2 sockets
Sep 27 03:30:31 dist-test-slave-j0hf krb5kdc[14014](info): commencing operation
krb5kdc: starting...
W20260927 03:30:32.363397 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:30:32 dist-test-slave-j0hf krb5kdc[14014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:30:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:30:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:30:32.480454 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43875
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43875
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:30:32.560623 14027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:30:32.560865 14027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:30:32.560902 14027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:30:32.562049 14027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:30:32.562114 14027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:30:32.562137 14027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:30:32.562166 14027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:30:32.562196 14027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:30:32.563675 14027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:30:32.563951 14027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:30:32.564184 14027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:30:32.569195 14033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:32.569355 14032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:32.569494 14027 server_base.cc:1034] running on GCE node
W20260927 03:30:32.569211 14035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:32.569806 14027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:30:32.570174 14027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:30:32.571375 14027 hybrid_clock.cc:648] HybridClock initialized: now 1790479832571360 us; error 29 us; skew 500 ppm
Sep 27 03:30:32 dist-test-slave-j0hf krb5kdc[14014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:30:32.574065 14027 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:30:32.574569 14027 webserver.cc:460] Webserver started at http://127.0.98.62:41793/ using document root <none> and password file <none>
I20260927 03:30:32.574762 14027 fs_manager.cc:359] Metadata directory not provided
I20260927 03:30:32.574806 14027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:30:32.574930 14027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:30:32.576032 14027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "8296a9b120124cc2bdcb964f17401d41"
format_stamp: "Formatted at 2026-09-27 03:30:32 on dist-test-slave-j0hf"
server_key: "4e62d0a469c2125eb2ca0bcaa6c70ba7"
server_key_iv: "55fec9194f65af6dff08d03ba9451a0b"
server_key_version: "encryptionkey@0"
I20260927 03:30:32.576339 14027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "8296a9b120124cc2bdcb964f17401d41"
format_stamp: "Formatted at 2026-09-27 03:30:32 on dist-test-slave-j0hf"
server_key: "4e62d0a469c2125eb2ca0bcaa6c70ba7"
server_key_iv: "55fec9194f65af6dff08d03ba9451a0b"
server_key_version: "encryptionkey@0"
I20260927 03:30:32.577630 14027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 03:30:32.578286 14042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:32.578460 14027 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:32.578547 14027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "8296a9b120124cc2bdcb964f17401d41"
format_stamp: "Formatted at 2026-09-27 03:30:32 on dist-test-slave-j0hf"
server_key: "4e62d0a469c2125eb2ca0bcaa6c70ba7"
server_key_iv: "55fec9194f65af6dff08d03ba9451a0b"
server_key_version: "encryptionkey@0"
I20260927 03:30:32.578600 14027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:30:32.600307 14027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:30:32.602193 14027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:30:32.602301 14027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:30:32.606123 14027 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43875
I20260927 03:30:32.606197 14094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43875 every 8 connection(s)
I20260927 03:30:32.606444 14027 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:30:32.607275 14095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:30:32.609635 14095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8296a9b120124cc2bdcb964f17401d41: Bootstrap starting.
I20260927 03:30:32.610227 14095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8296a9b120124cc2bdcb964f17401d41: Neither blocks nor log segments found. Creating new log.
I20260927 03:30:32.610515 14095 log.cc:824] T 00000000000000000000000000000000 P 8296a9b120124cc2bdcb964f17401d41: Log is configured to *not* fsync() on all Append() calls
I20260927 03:30:32.611222 14095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8296a9b120124cc2bdcb964f17401d41: No bootstrap required, opened a new log
I20260927 03:30:32.612838 14095 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8296a9b120124cc2bdcb964f17401d41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8296a9b120124cc2bdcb964f17401d41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43875 } }
I20260927 03:30:32.612974 14095 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8296a9b120124cc2bdcb964f17401d41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:30:32.612996 14095 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8296a9b120124cc2bdcb964f17401d41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8296a9b120124cc2bdcb964f17401d41, State: Initialized, Role: FOLLOWER
I20260927 03:30:32.613139 14095 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8296a9b120124cc2bdcb964f17401d41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8296a9b120124cc2bdcb964f17401d41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43875 } }
I20260927 03:30:32.613214 14095 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8296a9b120124cc2bdcb964f17401d41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:30:32.613252 14095 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8296a9b120124cc2bdcb964f17401d41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:30:32.613302 14095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8296a9b120124cc2bdcb964f17401d41 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:30:32.613828 14095 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8296a9b120124cc2bdcb964f17401d41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8296a9b120124cc2bdcb964f17401d41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43875 } }
I20260927 03:30:32.613934 14095 leader_election.cc:302] T 00000000000000000000000000000000 P 8296a9b120124cc2bdcb964f17401d41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8296a9b120124cc2bdcb964f17401d41; no voters:
I20260927 03:30:32.614115 14095 leader_election.cc:288] T 00000000000000000000000000000000 P 8296a9b120124cc2bdcb964f17401d41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:30:32.614194 14098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8296a9b120124cc2bdcb964f17401d41 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:30:32.614358 14095 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8296a9b120124cc2bdcb964f17401d41 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:30:32.614367 14098 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8296a9b120124cc2bdcb964f17401d41 [term 1 LEADER]: Becoming Leader. State: Replica: 8296a9b120124cc2bdcb964f17401d41, State: Running, Role: LEADER
I20260927 03:30:32.614595 14098 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8296a9b120124cc2bdcb964f17401d41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8296a9b120124cc2bdcb964f17401d41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43875 } }
I20260927 03:30:32.615013 14100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8296a9b120124cc2bdcb964f17401d41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8296a9b120124cc2bdcb964f17401d41. Latest consensus state: current_term: 1 leader_uuid: "8296a9b120124cc2bdcb964f17401d41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8296a9b120124cc2bdcb964f17401d41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43875 } } }
I20260927 03:30:32.615119 14100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8296a9b120124cc2bdcb964f17401d41 [sys.catalog]: This master's current role is: LEADER
I20260927 03:30:32.615140 14099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8296a9b120124cc2bdcb964f17401d41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8296a9b120124cc2bdcb964f17401d41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8296a9b120124cc2bdcb964f17401d41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43875 } } }
I20260927 03:30:32.615223 14099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8296a9b120124cc2bdcb964f17401d41 [sys.catalog]: This master's current role is: LEADER
I20260927 03:30:32.615453 14107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:30:32.615628 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 14027
I20260927 03:30:32.615741 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:30:32.616016 392 external_mini_cluster.cc:1428] Setting key 6448fa8e43e8387498e021e08ced218d
I20260927 03:30:32.616078 14107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:30:32.617830 14107 catalog_manager.cc:1347] Generated new cluster ID: ad975434bccd4ee4aca0d3b588b9cfd2
I20260927 03:30:32.617878 14107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:30:32 dist-test-slave-j0hf krb5kdc[14014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:30:32.627348 14118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:32.617561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55514 (local address 127.0.98.62:43875)
0927 03:30:32.617739 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:32.617748 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:32.617783 (+ 35us) server_negotiation.cc:407] Connection header received
0927 03:30:32.617906 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:32.617913 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:32.617984 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:32.618146 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:32.618833 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:32.619841 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:32.620648 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:32.621092 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:32.621228 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:32.623771 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:32.623776 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:32.623776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:32.623789 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:32.625630 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:32.626039 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:32.626042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:32.626043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:32.626153 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:32.626374 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:32.626375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:32.626376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:32.626638 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:32.626825 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:32.626927 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 03:30:32.627093 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260927 03:30:32.628484 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:30:32.634505 14107 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:30:32.635047 14107 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:30:32.638713 14107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8296a9b120124cc2bdcb964f17401d41: Generated new TSK 0
I20260927 03:30:32.638885 14107 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:30:32.682444 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 14027
2026-09-27T03:30:32Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1347 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:30:32 dist-test-slave-j0hf krb5kdc[14122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:30:32 dist-test-slave-j0hf krb5kdc[14122](info): set up 2 sockets
Sep 27 03:30:32 dist-test-slave-j0hf krb5kdc[14122](info): commencing operation
krb5kdc: starting...
W20260927 03:30:33.719341 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:30:33 dist-test-slave-j0hf krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:30:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:30:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:30:33.837786 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42247
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42247
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:30:33.919796 14135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:30:33.920013 14135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:30:33.920038 14135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:30:33.921202 14135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:30:33.921247 14135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:30:33.921262 14135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:30:33.921274 14135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:30:33.921286 14135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:30:33.922854 14135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:30:33.923115 14135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:30:33.923349 14135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:30:33.928212 14140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:33.928212 14143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:33.928272 14141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:33.928486 14135 server_base.cc:1034] running on GCE node
I20260927 03:30:33.928746 14135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:30:33.929039 14135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:30:33.930183 14135 hybrid_clock.cc:648] HybridClock initialized: now 1790479833930162 us; error 32 us; skew 500 ppm
Sep 27 03:30:33 dist-test-slave-j0hf krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:30:33.932826 14135 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:30:33.933270 14135 webserver.cc:460] Webserver started at http://127.0.98.62:34017/ using document root <none> and password file <none>
I20260927 03:30:33.933490 14135 fs_manager.cc:359] Metadata directory not provided
I20260927 03:30:33.933555 14135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:30:33.933712 14135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:30:33.934787 14135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "1eb88ac207a0431da5a2ab506e7c4439"
format_stamp: "Formatted at 2026-09-27 03:30:33 on dist-test-slave-j0hf"
server_key: "dab5835241f9ed84c17a1fc6f011a009"
server_key_iv: "3bbc06dab2bba2bb22de2cfab2bdf3db"
server_key_version: "encryptionkey@0"
I20260927 03:30:33.935140 14135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "1eb88ac207a0431da5a2ab506e7c4439"
format_stamp: "Formatted at 2026-09-27 03:30:33 on dist-test-slave-j0hf"
server_key: "dab5835241f9ed84c17a1fc6f011a009"
server_key_iv: "3bbc06dab2bba2bb22de2cfab2bdf3db"
server_key_version: "encryptionkey@0"
I20260927 03:30:33.936481 14135 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:30:33.937208 14150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:33.937424 14135 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:30:33.937494 14135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "1eb88ac207a0431da5a2ab506e7c4439"
format_stamp: "Formatted at 2026-09-27 03:30:33 on dist-test-slave-j0hf"
server_key: "dab5835241f9ed84c17a1fc6f011a009"
server_key_iv: "3bbc06dab2bba2bb22de2cfab2bdf3db"
server_key_version: "encryptionkey@0"
I20260927 03:30:33.937592 14135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:30:33.957974 14135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:30:33.959956 14135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:30:33.960120 14135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:30:33.963924 14135 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42247
I20260927 03:30:33.963968 14202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42247 every 8 connection(s)
I20260927 03:30:33.964332 14135 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:30:33.965260 14203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:30:33.967656 14203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439: Bootstrap starting.
I20260927 03:30:33.968222 14203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439: Neither blocks nor log segments found. Creating new log.
I20260927 03:30:33.968488 14203 log.cc:824] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439: Log is configured to *not* fsync() on all Append() calls
I20260927 03:30:33.969182 14203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439: No bootstrap required, opened a new log
I20260927 03:30:33.970763 14203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb88ac207a0431da5a2ab506e7c4439" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42247 } }
I20260927 03:30:33.970901 14203 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:30:33.970924 14203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb88ac207a0431da5a2ab506e7c4439, State: Initialized, Role: FOLLOWER
I20260927 03:30:33.971103 14203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb88ac207a0431da5a2ab506e7c4439" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42247 } }
I20260927 03:30:33.971185 14203 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:30:33.971226 14203 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:30:33.971274 14203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:30:33.972136 14203 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb88ac207a0431da5a2ab506e7c4439" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42247 } }
I20260927 03:30:33.972271 14203 leader_election.cc:302] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1eb88ac207a0431da5a2ab506e7c4439; no voters:
I20260927 03:30:33.972402 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 14135
I20260927 03:30:33.972442 14203 leader_election.cc:288] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:30:33.972491 14206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:30:33.972522 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:30:33.972646 14206 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [term 1 LEADER]: Becoming Leader. State: Replica: 1eb88ac207a0431da5a2ab506e7c4439, State: Running, Role: LEADER
I20260927 03:30:33.972714 14203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:30:33.972759 392 external_mini_cluster.cc:1428] Setting key f09fa9786bd3c7aeeb5035ecda3b8a23
I20260927 03:30:33.972795 14206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb88ac207a0431da5a2ab506e7c4439" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42247 } }
I20260927 03:30:33.973111 14207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1eb88ac207a0431da5a2ab506e7c4439" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb88ac207a0431da5a2ab506e7c4439" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42247 } } }
I20260927 03:30:33.973181 14207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [sys.catalog]: This master's current role is: LEADER
I20260927 03:30:33.973376 14208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1eb88ac207a0431da5a2ab506e7c4439. Latest consensus state: current_term: 1 leader_uuid: "1eb88ac207a0431da5a2ab506e7c4439" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb88ac207a0431da5a2ab506e7c4439" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42247 } } }
I20260927 03:30:33.973631 14208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [sys.catalog]: This master's current role is: LEADER
I20260927 03:30:33.973973 14211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:30:33.974488 14211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:30:33.976076 14211 catalog_manager.cc:1347] Generated new cluster ID: b4222c1023a74777b1e31fd182888bb6
I20260927 03:30:33.976132 14211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:30:33 dist-test-slave-j0hf krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:30:33.984728 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:33.973396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54152 (local address 127.0.98.62:42247)
0927 03:30:33.973918 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:33.973928 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:33.973964 (+ 36us) server_negotiation.cc:407] Connection header received
0927 03:30:33.974084 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:33.974091 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:33.974145 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:33.974297 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:33.975210 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:33.976531 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:33.977544 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:33.978224 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:33.978340 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:33.981087 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:33.981091 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:33.981091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:33.981103 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:33.983004 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:33.983451 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:33.983454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:33.983454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:33.983576 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:33.983783 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:33.983784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:33.983784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:33.984062 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:33.984213 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:33.984324 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 03:30:33.984484 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":222,"threads_started":1}
I20260927 03:30:33.987226 14211 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:30:33.987914 14211 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:30:33.993530 14211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439: Generated new TSK 0
I20260927 03:30:33.993717 14211 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:30:34.020613 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42247
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:30:34.103857 14229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:30:34.104092 14229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:30:34.104118 14229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:30:34.105387 14229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:30:34.105453 14229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:30:34.105509 14229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:30:34.107085 14229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:30:34.107339 14229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:30:34.107604 14229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:30:34.112691 14235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:34.112748 14234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:34.112694 14237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:34.112950 14229 server_base.cc:1034] running on GCE node
I20260927 03:30:34.113154 14229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:30:34.113404 14229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:30:34.114562 14229 hybrid_clock.cc:648] HybridClock initialized: now 1790479834114555 us; error 38 us; skew 500 ppm
Sep 27 03:30:34 dist-test-slave-j0hf krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:30:34.117254 14229 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:30:34.117723 14229 webserver.cc:460] Webserver started at http://127.0.98.1:44629/ using document root <none> and password file <none>
I20260927 03:30:34.117954 14229 fs_manager.cc:359] Metadata directory not provided
I20260927 03:30:34.118019 14229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:30:34.118177 14229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:30:34.119248 14229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/data/instance:
uuid: "1616528b7a99454abb6205c58f44d4e9"
format_stamp: "Formatted at 2026-09-27 03:30:34 on dist-test-slave-j0hf"
server_key: "e9746c620ac559e974335a81cb02a692"
server_key_iv: "399eca5579795a64b969cf5c5bd745cf"
server_key_version: "encryptionkey@0"
I20260927 03:30:34.119633 14229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1616528b7a99454abb6205c58f44d4e9"
format_stamp: "Formatted at 2026-09-27 03:30:34 on dist-test-slave-j0hf"
server_key: "e9746c620ac559e974335a81cb02a692"
server_key_iv: "399eca5579795a64b969cf5c5bd745cf"
server_key_version: "encryptionkey@0"
I20260927 03:30:34.120957 14229 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:30:34.121724 14244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:34.121922 14229 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:30:34.122017 14229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "1616528b7a99454abb6205c58f44d4e9"
format_stamp: "Formatted at 2026-09-27 03:30:34 on dist-test-slave-j0hf"
server_key: "e9746c620ac559e974335a81cb02a692"
server_key_iv: "399eca5579795a64b969cf5c5bd745cf"
server_key_version: "encryptionkey@0"
I20260927 03:30:34.122097 14229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:30:34.132331 14229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:30:34.134629 14229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:30:34.134792 14229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:30:34.135028 14229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:30:34.135398 14229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:30:34.135478 14229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:34.135527 14229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:30:34.135553 14229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:34.141782 14229 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46393
I20260927 03:30:34.141832 14357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46393 every 8 connection(s)
I20260927 03:30:34.142153 14229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:30:34.145478 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 14229
I20260927 03:30:34.145579 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:30:34.145879 392 external_mini_cluster.cc:1428] Setting key c35e464820ef73c35e1970abe1288cb8
Sep 27 03:30:34 dist-test-slave-j0hf krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:30:34.152942 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:34.142854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53295 (local address 127.0.98.62:42247)
0927 03:30:34.142922 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:34.142925 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:34.143133 (+ 208us) server_negotiation.cc:407] Connection header received
0927 03:30:34.143702 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:34.143703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:34.143717 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:34.143772 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:34.144706 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.145209 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.146543 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.146789 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.146881 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:34.149465 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:34.149467 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:34.149467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:34.149475 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:34.150994 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:34.151733 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:34.151735 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:34.151735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:34.151792 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:34.152199 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:34.152200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:34.152201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:34.152346 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:34.152431 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:34.152737 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 03:30:34.152847 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 03:30:34.153131 14360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:34.142850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.1:53295)
0927 03:30:34.143019 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:34.143056 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:34.143614 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:34.143844 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:34.143847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:34.144420 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:30:34.144626 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:34.144631 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.145285 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.145286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:34.146435 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:34.146439 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.146893 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.146894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:34.147028 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:34.147795 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:30:34.147804 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:30:34.149362 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:30:34.151105 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:34.151112 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:34.151115 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:34.151618 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:34.151871 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:34.151872 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:34.151872 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:34.152092 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:34.152422 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:30:34.152429 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:30:34.152582 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 03:30:34.152699 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 03:30:34.152867 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
I20260927 03:30:34.153824 14358 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42247
I20260927 03:30:34.153927 14358 heartbeater.cc:464] Registering TS with master...
I20260927 03:30:34.154125 14358 heartbeater.cc:511] Master 127.0.98.62:42247 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 03:30:34.154737 14167 ts_manager.cc:194] Registered new tserver with Master: 1616528b7a99454abb6205c58f44d4e9 (127.0.98.1:46393)
I20260927 03:30:34.155354 14167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53295
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:30:34.174268 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42247
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:30:34.260443 14363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:30:34.260723 14363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:30:34.260763 14363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:30:34.262210 14363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:30:34.262259 14363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:30:34.262285 14363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:30:34.264061 14363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:30:34.264335 14363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:30:34.264595 14363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:30:34.269721 14368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:34.269742 14369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:34.269721 14371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:34.269757 14363 server_base.cc:1034] running on GCE node
I20260927 03:30:34.270215 14363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:30:34.270460 14363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:30:34.271625 14363 hybrid_clock.cc:648] HybridClock initialized: now 1790479834271611 us; error 26 us; skew 500 ppm
Sep 27 03:30:34 dist-test-slave-j0hf krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:30:34.274435 14363 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:30:34.274863 14363 webserver.cc:460] Webserver started at http://127.0.98.2:37079/ using document root <none> and password file <none>
I20260927 03:30:34.275046 14363 fs_manager.cc:359] Metadata directory not provided
I20260927 03:30:34.275089 14363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:30:34.275177 14363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:30:34.276257 14363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/data/instance:
uuid: "63bd9fc884d64ba6a55e09100051defc"
format_stamp: "Formatted at 2026-09-27 03:30:34 on dist-test-slave-j0hf"
server_key: "07f780196cf5073bfea103fbd09d8ee6"
server_key_iv: "0a1b79957bcffafaac6859c4313eb226"
server_key_version: "encryptionkey@0"
I20260927 03:30:34.276575 14363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/wal/instance:
uuid: "63bd9fc884d64ba6a55e09100051defc"
format_stamp: "Formatted at 2026-09-27 03:30:34 on dist-test-slave-j0hf"
server_key: "07f780196cf5073bfea103fbd09d8ee6"
server_key_iv: "0a1b79957bcffafaac6859c4313eb226"
server_key_version: "encryptionkey@0"
I20260927 03:30:34.278167 14363 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:30:34.278920 14378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:34.279168 14363 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:30:34.279277 14363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "63bd9fc884d64ba6a55e09100051defc"
format_stamp: "Formatted at 2026-09-27 03:30:34 on dist-test-slave-j0hf"
server_key: "07f780196cf5073bfea103fbd09d8ee6"
server_key_iv: "0a1b79957bcffafaac6859c4313eb226"
server_key_version: "encryptionkey@0"
I20260927 03:30:34.279371 14363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:30:34.294839 14363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:30:34.297586 14363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:30:34.297724 14363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:30:34.297935 14363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:30:34.298307 14363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:30:34.298349 14363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:34.298403 14363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:30:34.298430 14363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:34.304723 14363 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39885
I20260927 03:30:34.304802 14491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39885 every 8 connection(s)
I20260927 03:30:34.305177 14363 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:30:34.309847 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 14363
I20260927 03:30:34.309944 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:30:34.310571 392 external_mini_cluster.cc:1428] Setting key 2dddaa3346df2d11d48b29d1fab7a4cc
Sep 27 03:30:34 dist-test-slave-j0hf krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:30:34.315549 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:34.305956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37219 (local address 127.0.98.62:42247)
0927 03:30:34.306046 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:34.306048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:34.306240 (+ 192us) server_negotiation.cc:407] Connection header received
0927 03:30:34.306768 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:34.306769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:34.306783 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:34.306830 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:34.307846 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.308362 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.309354 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.309575 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.309662 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:34.312314 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:34.312315 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:34.312316 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:34.312323 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:34.314087 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:34.314506 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:34.314515 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:34.314516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:34.314579 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:34.314855 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:34.314856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:34.314856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:34.314978 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:34.315060 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:34.315338 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 03:30:34.315449 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 03:30:34.315755 14494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:34.305911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.2:37219)
0927 03:30:34.306110 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:34.306148 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:34.306658 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:34.306909 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:34.306913 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:34.307542 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:30:34.307741 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:34.307746 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.308454 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.308456 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:34.309263 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:34.309265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.309671 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.309672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:34.309796 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:34.310380 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:30:34.310391 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:30:34.312212 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:30:34.314183 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:34.314191 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:34.314193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:34.314426 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:34.314648 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:34.314649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:34.314649 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:34.314779 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:34.315049 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:30:34.315055 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:30:34.315205 (+ 150us) client_negotiation.cc:769] Sending connection context
0927 03:30:34.315319 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 03:30:34.315501 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":107,"threads_started":1}
I20260927 03:30:34.316229 14492 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42247
I20260927 03:30:34.316358 14492 heartbeater.cc:464] Registering TS with master...
I20260927 03:30:34.316588 14492 heartbeater.cc:511] Master 127.0.98.62:42247 requested a full tablet report, sending...
I20260927 03:30:34.317005 14167 ts_manager.cc:194] Registered new tserver with Master: 63bd9fc884d64ba6a55e09100051defc (127.0.98.2:39885)
I20260927 03:30:34.317435 14167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37219
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:30:34.338500 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42247
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:30:34.418844 14497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:30:34.419049 14497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:30:34.419073 14497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:30:34.420177 14497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:30:34.420209 14497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:30:34.420234 14497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:30:34.421561 14497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:30:34.421732 14497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:30:34.421901 14497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:30:34.426837 14505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:34.426893 14497 server_base.cc:1034] running on GCE node
W20260927 03:30:34.426992 14502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:34.426857 14503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:34.427227 14497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:30:34.427533 14497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:30:34.428699 14497 hybrid_clock.cc:648] HybridClock initialized: now 1790479834428676 us; error 40 us; skew 500 ppm
Sep 27 03:30:34 dist-test-slave-j0hf krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:30:34.431164 14497 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:30:34.431674 14497 webserver.cc:460] Webserver started at http://127.0.98.3:38203/ using document root <none> and password file <none>
I20260927 03:30:34.431900 14497 fs_manager.cc:359] Metadata directory not provided
I20260927 03:30:34.431968 14497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:30:34.432119 14497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:30:34.433153 14497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/data/instance:
uuid: "1d48129deb7b4c02b81063b6d9ebc562"
format_stamp: "Formatted at 2026-09-27 03:30:34 on dist-test-slave-j0hf"
server_key: "d466fd0bfc3ea9b165c5dfcad1a18e93"
server_key_iv: "ac2decac81bd1c7a4ed079e0ca8ffb83"
server_key_version: "encryptionkey@0"
I20260927 03:30:34.433522 14497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1d48129deb7b4c02b81063b6d9ebc562"
format_stamp: "Formatted at 2026-09-27 03:30:34 on dist-test-slave-j0hf"
server_key: "d466fd0bfc3ea9b165c5dfcad1a18e93"
server_key_iv: "ac2decac81bd1c7a4ed079e0ca8ffb83"
server_key_version: "encryptionkey@0"
I20260927 03:30:34.434823 14497 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:30:34.435606 14512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:34.435787 14497 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:30:34.435882 14497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "1d48129deb7b4c02b81063b6d9ebc562"
format_stamp: "Formatted at 2026-09-27 03:30:34 on dist-test-slave-j0hf"
server_key: "d466fd0bfc3ea9b165c5dfcad1a18e93"
server_key_iv: "ac2decac81bd1c7a4ed079e0ca8ffb83"
server_key_version: "encryptionkey@0"
I20260927 03:30:34.435969 14497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:30:34.445919 14497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:30:34.447955 14497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:30:34.448124 14497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:30:34.448376 14497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:30:34.448694 14497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:30:34.448748 14497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:34.448791 14497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:30:34.448819 14497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:34.454483 14497 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38037
I20260927 03:30:34.454581 14625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38037 every 8 connection(s)
I20260927 03:30:34.454892 14497 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:30:34 dist-test-slave-j0hf krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:30:34.463631 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 14497
I20260927 03:30:34.463730 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:30:34.463936 392 external_mini_cluster.cc:1428] Setting key fe4cd721d614839b4feff5e0fb8ba4b9
I20260927 03:30:34.464082 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:34.455716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44163 (local address 127.0.98.62:42247)
0927 03:30:34.455783 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:34.455785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:34.455981 (+ 196us) server_negotiation.cc:407] Connection header received
0927 03:30:34.456545 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:34.456547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:34.456567 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:34.456614 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:34.457598 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.458103 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.459145 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.459347 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.459424 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:34.461613 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:34.461615 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:34.461615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:34.461622 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:34.462757 (+ 1135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:34.463092 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:34.463093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:34.463093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:34.463140 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:34.463339 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:34.463339 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:34.463339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:34.463498 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:34.463559 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:34.463896 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 03:30:34.463957 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 03:30:34.464290 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:34.455699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.3:44163)
0927 03:30:34.455860 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:34.455894 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:34.456437 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:34.456699 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:34.456703 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:34.457283 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:30:34.457475 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:34.457479 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.458195 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.458196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:34.459043 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:34.459045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.459457 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.459458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:34.459582 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:34.460135 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:30:34.460143 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:30:34.461496 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:30:34.462824 (+ 1328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:34.462829 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:34.462830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:34.463028 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:34.463195 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:34.463195 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:34.463196 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:34.463287 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:34.463579 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:30:34.463586 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:30:34.463754 (+ 168us) client_negotiation.cc:769] Sending connection context
0927 03:30:34.463880 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 03:30:34.464042 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260927 03:30:34.464694 14626 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42247
I20260927 03:30:34.464799 14626 heartbeater.cc:464] Registering TS with master...
I20260927 03:30:34.464972 14626 heartbeater.cc:511] Master 127.0.98.62:42247 requested a full tablet report, sending...
I20260927 03:30:34.465303 14167 ts_manager.cc:194] Registered new tserver with Master: 1d48129deb7b4c02b81063b6d9ebc562 (127.0.98.3:38037)
I20260927 03:30:34.465670 14167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44163
I20260927 03:30:34.475785 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:30:34.483237 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:34.477159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54156 (local address 127.0.98.62:42247)
0927 03:30:34.477202 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:34.477204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:34.477455 (+ 251us) server_negotiation.cc:407] Connection header received
0927 03:30:34.477510 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:34.477511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:34.477525 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:34.477576 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:34.478127 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.478596 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.479396 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.479650 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.479712 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:34.480901 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:34.480903 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:34.480903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:34.480909 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:34.482162 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:34.482547 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:34.482548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:34.482548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:34.482597 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:34.482808 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:34.482808 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:34.482808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:34.482937 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:34.483011 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:34.483112 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 03:30:34.483171 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:30:34.484915 14167 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:30:34.486476 14167 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:30:34.493193 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:34.488698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46393 (local address 127.0.0.1:41760)
0927 03:30:34.489291 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:34.489300 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:34.489360 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:34.489756 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:34.489756 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:34.489759 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:30:34.489803 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:34.489804 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.490697 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.490698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:34.492182 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:34.492184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.492887 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.492888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:34.492969 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:34.492982 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 03:30:34.493054 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 03:30:34.493113 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":65,"thread_start_us":59,"threads_started":1}
I20260927 03:30:34.493208 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:34.488824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41760 (local address 127.0.98.1:46393)
0927 03:30:34.489262 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:34.489267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:34.489346 (+ 79us) server_negotiation.cc:407] Connection header received
0927 03:30:34.489639 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:34.489643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:34.489659 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:34.489762 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:30:34.489854 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.490592 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.492282 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.492783 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.492883 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:34.492946 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:34.493075 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 03:30:34.493120 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":104,"threads_started":1}
I20260927 03:30:34.493403 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:34.489058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35558 (local address 127.0.98.2:39885)
0927 03:30:34.489243 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:34.489247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:34.489537 (+ 290us) server_negotiation.cc:407] Connection header received
0927 03:30:34.489693 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:34.489695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:34.489712 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:34.489780 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:30:34.489995 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.490980 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.492131 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.492885 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.492964 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:34.493022 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:34.493283 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 03:30:34.493328 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":81,"threads_started":1}
I20260927 03:30:34.493693 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:34.489033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45964 (local address 127.0.98.3:38037)
0927 03:30:34.489237 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:34.489242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:34.489251 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:30:34.489287 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:34.489290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:34.489310 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:34.489396 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:30:34.489611 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.490722 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.492268 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.492914 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.492972 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:34.493022 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:34.493558 (+ 536us) server_negotiation.cc:299] Negotiation successful
0927 03:30:34.493611 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":93,"threads_started":1}
I20260927 03:30:34.493877 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:34.488698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38037 (local address 127.0.0.1:45964)
0927 03:30:34.489028 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:34.489070 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:34.489144 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:34.489401 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:34.489404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:34.489407 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:30:34.489477 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:34.489479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.490822 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.490823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:34.492182 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:34.492184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.493062 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.493063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:34.493134 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:34.493144 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 03:30:34.493205 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 03:30:34.493238 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":98,"thread_start_us":63,"threads_started":1}
I20260927 03:30:34.494264 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:34.488899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39885 (local address 127.0.0.1:35558)
0927 03:30:34.489468 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:34.489477 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:34.489541 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:34.489860 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:34.489861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:34.489863 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:30:34.489916 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:34.489917 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.491074 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.491075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:34.492039 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:34.492042 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.493060 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.493062 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:34.493134 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:34.493142 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 03:30:34.493194 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 03:30:34.493238 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":5,"thread_start_us":67,"threads_started":1}
I20260927 03:30:34.494253 14292 tablet_service.cc:1467] Processing CreateTablet for tablet 5755bc931afd44c78f949d350dac8f54 (DEFAULT_TABLE table=test-table [id=8a9641e70dbe4e12ad8def716535965e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:30:34.494639 14292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5755bc931afd44c78f949d350dac8f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:30:34.494577 14560 tablet_service.cc:1467] Processing CreateTablet for tablet 5755bc931afd44c78f949d350dac8f54 (DEFAULT_TABLE table=test-table [id=8a9641e70dbe4e12ad8def716535965e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:30:34.494896 14560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5755bc931afd44c78f949d350dac8f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:30:34.496117 14426 tablet_service.cc:1467] Processing CreateTablet for tablet 5755bc931afd44c78f949d350dac8f54 (DEFAULT_TABLE table=test-table [id=8a9641e70dbe4e12ad8def716535965e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:30:34.496469 14426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5755bc931afd44c78f949d350dac8f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:30:34.497610 14640 tablet_bootstrap.cc:492] T 5755bc931afd44c78f949d350dac8f54 P 1616528b7a99454abb6205c58f44d4e9: Bootstrap starting.
I20260927 03:30:34.498171 14641 tablet_bootstrap.cc:492] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562: Bootstrap starting.
I20260927 03:30:34.498397 14640 tablet_bootstrap.cc:654] T 5755bc931afd44c78f949d350dac8f54 P 1616528b7a99454abb6205c58f44d4e9: Neither blocks nor log segments found. Creating new log.
I20260927 03:30:34.498874 14641 tablet_bootstrap.cc:654] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562: Neither blocks nor log segments found. Creating new log.
I20260927 03:30:34.499109 14640 log.cc:824] T 5755bc931afd44c78f949d350dac8f54 P 1616528b7a99454abb6205c58f44d4e9: Log is configured to *not* fsync() on all Append() calls
I20260927 03:30:34.499497 14641 log.cc:824] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562: Log is configured to *not* fsync() on all Append() calls
I20260927 03:30:34.499861 14643 tablet_bootstrap.cc:492] T 5755bc931afd44c78f949d350dac8f54 P 63bd9fc884d64ba6a55e09100051defc: Bootstrap starting.
I20260927 03:30:34.500030 14640 tablet_bootstrap.cc:492] T 5755bc931afd44c78f949d350dac8f54 P 1616528b7a99454abb6205c58f44d4e9: No bootstrap required, opened a new log
I20260927 03:30:34.500108 14640 ts_tablet_manager.cc:1397] T 5755bc931afd44c78f949d350dac8f54 P 1616528b7a99454abb6205c58f44d4e9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 03:30:34.500255 14641 tablet_bootstrap.cc:492] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562: No bootstrap required, opened a new log
I20260927 03:30:34.500329 14641 ts_tablet_manager.cc:1397] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:30:34.500540 14643 tablet_bootstrap.cc:654] T 5755bc931afd44c78f949d350dac8f54 P 63bd9fc884d64ba6a55e09100051defc: Neither blocks nor log segments found. Creating new log.
I20260927 03:30:34.501070 14643 log.cc:824] T 5755bc931afd44c78f949d350dac8f54 P 63bd9fc884d64ba6a55e09100051defc: Log is configured to *not* fsync() on all Append() calls
I20260927 03:30:34.501868 14643 tablet_bootstrap.cc:492] T 5755bc931afd44c78f949d350dac8f54 P 63bd9fc884d64ba6a55e09100051defc: No bootstrap required, opened a new log
I20260927 03:30:34.501942 14643 ts_tablet_manager.cc:1397] T 5755bc931afd44c78f949d350dac8f54 P 63bd9fc884d64ba6a55e09100051defc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:30:34.501947 14640 raft_consensus.cc:348] T 5755bc931afd44c78f949d350dac8f54 P 1616528b7a99454abb6205c58f44d4e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d48129deb7b4c02b81063b6d9ebc562" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38037 } } peers { permanent_uuid: "1616528b7a99454abb6205c58f44d4e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46393 } } peers { permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 } }
I20260927 03:30:34.502142 14640 raft_consensus.cc:374] T 5755bc931afd44c78f949d350dac8f54 P 1616528b7a99454abb6205c58f44d4e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:30:34.502185 14640 raft_consensus.cc:729] T 5755bc931afd44c78f949d350dac8f54 P 1616528b7a99454abb6205c58f44d4e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1616528b7a99454abb6205c58f44d4e9, State: Initialized, Role: FOLLOWER
I20260927 03:30:34.502298 14640 consensus_queue.cc:260] T 5755bc931afd44c78f949d350dac8f54 P 1616528b7a99454abb6205c58f44d4e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d48129deb7b4c02b81063b6d9ebc562" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38037 } } peers { permanent_uuid: "1616528b7a99454abb6205c58f44d4e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46393 } } peers { permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 } }
I20260927 03:30:34.502594 14640 ts_tablet_manager.cc:1428] T 5755bc931afd44c78f949d350dac8f54 P 1616528b7a99454abb6205c58f44d4e9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 03:30:34.502731 14358 heartbeater.cc:503] Master 127.0.98.62:42247 was elected leader, sending a full tablet report...
I20260927 03:30:34.502765 14641 raft_consensus.cc:348] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d48129deb7b4c02b81063b6d9ebc562" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38037 } } peers { permanent_uuid: "1616528b7a99454abb6205c58f44d4e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46393 } } peers { permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 } }
I20260927 03:30:34.502916 14641 raft_consensus.cc:374] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:30:34.502949 14641 raft_consensus.cc:729] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d48129deb7b4c02b81063b6d9ebc562, State: Initialized, Role: FOLLOWER
I20260927 03:30:34.503072 14641 consensus_queue.cc:260] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d48129deb7b4c02b81063b6d9ebc562" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38037 } } peers { permanent_uuid: "1616528b7a99454abb6205c58f44d4e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46393 } } peers { permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 } }
I20260927 03:30:34.503347 14626 heartbeater.cc:503] Master 127.0.98.62:42247 was elected leader, sending a full tablet report...
I20260927 03:30:34.503338 14641 ts_tablet_manager.cc:1428] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:30:34.504020 14643 raft_consensus.cc:348] T 5755bc931afd44c78f949d350dac8f54 P 63bd9fc884d64ba6a55e09100051defc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d48129deb7b4c02b81063b6d9ebc562" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38037 } } peers { permanent_uuid: "1616528b7a99454abb6205c58f44d4e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46393 } } peers { permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 } }
I20260927 03:30:34.504225 14643 raft_consensus.cc:374] T 5755bc931afd44c78f949d350dac8f54 P 63bd9fc884d64ba6a55e09100051defc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:30:34.504274 14643 raft_consensus.cc:729] T 5755bc931afd44c78f949d350dac8f54 P 63bd9fc884d64ba6a55e09100051defc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63bd9fc884d64ba6a55e09100051defc, State: Initialized, Role: FOLLOWER
I20260927 03:30:34.504379 14643 consensus_queue.cc:260] T 5755bc931afd44c78f949d350dac8f54 P 63bd9fc884d64ba6a55e09100051defc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d48129deb7b4c02b81063b6d9ebc562" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38037 } } peers { permanent_uuid: "1616528b7a99454abb6205c58f44d4e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46393 } } peers { permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 } }
I20260927 03:30:34.504593 14643 ts_tablet_manager.cc:1428] T 5755bc931afd44c78f949d350dac8f54 P 63bd9fc884d64ba6a55e09100051defc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:30:34.504644 14492 heartbeater.cc:503] Master 127.0.98.62:42247 was elected leader, sending a full tablet report...
W20260927 03:30:34.555850 14493 tablet.cc:2369] T 5755bc931afd44c78f949d350dac8f54 P 63bd9fc884d64ba6a55e09100051defc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:30:34.620752 14647 raft_consensus.cc:482] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:30:34.620890 14647 raft_consensus.cc:504] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d48129deb7b4c02b81063b6d9ebc562" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38037 } } peers { permanent_uuid: "1616528b7a99454abb6205c58f44d4e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46393 } } peers { permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 } }
I20260927 03:30:34.621460 14647 leader_election.cc:288] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1616528b7a99454abb6205c58f44d4e9 (127.0.98.1:46393), 63bd9fc884d64ba6a55e09100051defc (127.0.98.2:39885)
I20260927 03:30:34.624548 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:34.621792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58225 (local address 127.0.98.2:39885)
0927 03:30:34.621836 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:34.621838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:34.621849 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:30:34.621910 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:34.621910 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:34.621923 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:34.621980 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:30:34.622150 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.622620 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.623761 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.624173 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.624271 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:34.624282 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:34.624436 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 03:30:34.624475 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:30:34.624569 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:34.621811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53811 (local address 127.0.98.1:46393)
0927 03:30:34.621859 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:34.621862 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:34.621960 (+ 98us) server_negotiation.cc:407] Connection header received
0927 03:30:34.622043 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:34.622044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:34.622061 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:34.622124 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:30:34.622288 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.622741 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.623789 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.624219 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.624300 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:34.624310 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:34.624460 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 03:30:34.624496 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:30:34.624591 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:34.621671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39885 (local address 127.0.98.3:58225)
0927 03:30:34.621757 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:34.621772 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:34.621843 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:34.621994 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:34.621996 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:34.621997 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:30:34.622073 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:34.622075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.622712 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.622714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:34.623673 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:34.623676 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.624273 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.624273 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:34.624352 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:34.624355 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:30:34.624428 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 03:30:34.624471 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":32}
I20260927 03:30:34.624606 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:34.621688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46393 (local address 127.0.98.3:53811)
0927 03:30:34.621869 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:34.621879 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:34.621972 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:34.622129 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:34.622130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:34.622131 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:30:34.622215 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:34.622217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.622832 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.622834 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:34.623696 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:34.623699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.624302 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.624303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:34.624379 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:34.624382 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:30:34.624453 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 03:30:34.624492 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":108,"threads_started":1}
I20260927 03:30:34.625016 14446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5755bc931afd44c78f949d350dac8f54" candidate_uuid: "1d48129deb7b4c02b81063b6d9ebc562" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63bd9fc884d64ba6a55e09100051defc" is_pre_election: true
I20260927 03:30:34.625217 14446 raft_consensus.cc:2454] T 5755bc931afd44c78f949d350dac8f54 P 63bd9fc884d64ba6a55e09100051defc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d48129deb7b4c02b81063b6d9ebc562 in term 0.
I20260927 03:30:34.625200 14312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5755bc931afd44c78f949d350dac8f54" candidate_uuid: "1d48129deb7b4c02b81063b6d9ebc562" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1616528b7a99454abb6205c58f44d4e9" is_pre_election: true
I20260927 03:30:34.625380 14312 raft_consensus.cc:2454] T 5755bc931afd44c78f949d350dac8f54 P 1616528b7a99454abb6205c58f44d4e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d48129deb7b4c02b81063b6d9ebc562 in term 0.
I20260927 03:30:34.625445 14514 leader_election.cc:302] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d48129deb7b4c02b81063b6d9ebc562, 63bd9fc884d64ba6a55e09100051defc; no voters:
I20260927 03:30:34.625594 14647 raft_consensus.cc:2781] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:30:34.625667 14647 raft_consensus.cc:482] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:30:34.625723 14647 raft_consensus.cc:3037] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:30:34.626427 14647 raft_consensus.cc:504] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d48129deb7b4c02b81063b6d9ebc562" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38037 } } peers { permanent_uuid: "1616528b7a99454abb6205c58f44d4e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46393 } } peers { permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 } }
I20260927 03:30:34.626636 14647 leader_election.cc:288] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562 [CANDIDATE]: Term 1 election: Requested vote from peers 1616528b7a99454abb6205c58f44d4e9 (127.0.98.1:46393), 63bd9fc884d64ba6a55e09100051defc (127.0.98.2:39885)
I20260927 03:30:34.626710 14312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5755bc931afd44c78f949d350dac8f54" candidate_uuid: "1d48129deb7b4c02b81063b6d9ebc562" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1616528b7a99454abb6205c58f44d4e9"
I20260927 03:30:34.626696 14446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5755bc931afd44c78f949d350dac8f54" candidate_uuid: "1d48129deb7b4c02b81063b6d9ebc562" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63bd9fc884d64ba6a55e09100051defc"
I20260927 03:30:34.626770 14446 raft_consensus.cc:3037] T 5755bc931afd44c78f949d350dac8f54 P 63bd9fc884d64ba6a55e09100051defc [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:30:34.626770 14312 raft_consensus.cc:3037] T 5755bc931afd44c78f949d350dac8f54 P 1616528b7a99454abb6205c58f44d4e9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:30:34.627686 14446 raft_consensus.cc:2454] T 5755bc931afd44c78f949d350dac8f54 P 63bd9fc884d64ba6a55e09100051defc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d48129deb7b4c02b81063b6d9ebc562 in term 1.
I20260927 03:30:34.627686 14312 raft_consensus.cc:2454] T 5755bc931afd44c78f949d350dac8f54 P 1616528b7a99454abb6205c58f44d4e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d48129deb7b4c02b81063b6d9ebc562 in term 1.
I20260927 03:30:34.627882 14514 leader_election.cc:302] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d48129deb7b4c02b81063b6d9ebc562, 63bd9fc884d64ba6a55e09100051defc; no voters:
I20260927 03:30:34.627977 14647 raft_consensus.cc:2781] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:30:34.628166 14647 raft_consensus.cc:686] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562 [term 1 LEADER]: Becoming Leader. State: Replica: 1d48129deb7b4c02b81063b6d9ebc562, State: Running, Role: LEADER
I20260927 03:30:34.628284 14647 consensus_queue.cc:237] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d48129deb7b4c02b81063b6d9ebc562" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38037 } } peers { permanent_uuid: "1616528b7a99454abb6205c58f44d4e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46393 } } peers { permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 } }
I20260927 03:30:34.629163 14167 catalog_manager.cc:5495] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d48129deb7b4c02b81063b6d9ebc562 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1d48129deb7b4c02b81063b6d9ebc562" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d48129deb7b4c02b81063b6d9ebc562" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1616528b7a99454abb6205c58f44d4e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 } health_report { overall_health: UNKNOWN } } }
I20260927 03:30:34.638723 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:34.636555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45972 (local address 127.0.98.3:38037)
0927 03:30:34.636602 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:34.636604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:34.636611 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:30:34.636637 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:34.636638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:34.636653 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:34.636711 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:30:34.636837 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.637294 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.638039 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.638247 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.638297 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:34.638441 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:30:34.638530 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:30:34.638589 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:34.638605 (+ 16us) server_negotiation.cc:299] Negotiation successful
0927 03:30:34.638621 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 03:30:34.640530 14446 raft_consensus.cc:1261] T 5755bc931afd44c78f949d350dac8f54 P 63bd9fc884d64ba6a55e09100051defc [term 1 FOLLOWER]: Refusing update from remote peer 1d48129deb7b4c02b81063b6d9ebc562: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:30:34.640556 14312 raft_consensus.cc:1261] T 5755bc931afd44c78f949d350dac8f54 P 1616528b7a99454abb6205c58f44d4e9 [term 1 FOLLOWER]: Refusing update from remote peer 1d48129deb7b4c02b81063b6d9ebc562: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:30:34.641058 14650 consensus_queue.cc:1035] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1616528b7a99454abb6205c58f44d4e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:30:34.641228 14647 consensus_queue.cc:1035] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 03:30:34.642853 14359 tablet.cc:2369] T 5755bc931afd44c78f949d350dac8f54 P 1616528b7a99454abb6205c58f44d4e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:30:34.643532 14653 mvcc.cc:205] Tried to move back new op lower bound from 7333805402684792832 to 7333805402638024704. Current Snapshot: MvccSnapshot[applied={T|T < 7333805402684792832}]
I20260927 03:30:34.643751 14654 mvcc.cc:205] Tried to move back new op lower bound from 7333805402684792832 to 7333805402638024704. Current Snapshot: MvccSnapshot[applied={T|T < 7333805402684792832}]
I20260927 03:30:34.644250 14655 mvcc.cc:205] Tried to move back new op lower bound from 7333805402684792832 to 7333805402638024704. Current Snapshot: MvccSnapshot[applied={T|T < 7333805402684792832}]
I20260927 03:30:34.646303 14167 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:30:34.646382 14167 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:30:34.647162 14167 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439: Sending DeleteTablet for 3 replicas of tablet 5755bc931afd44c78f949d350dac8f54
I20260927 03:30:34.647516 14560 tablet_service.cc:1514] Processing DeleteTablet for tablet 5755bc931afd44c78f949d350dac8f54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:30:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45964
I20260927 03:30:34.647575 14426 tablet_service.cc:1514] Processing DeleteTablet for tablet 5755bc931afd44c78f949d350dac8f54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:30:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35558
I20260927 03:30:34.647585 14292 tablet_service.cc:1514] Processing DeleteTablet for tablet 5755bc931afd44c78f949d350dac8f54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:30:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41760
I20260927 03:30:34.647789 14663 tablet_replica.cc:330] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562: stopping tablet replica
I20260927 03:30:34.647828 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 14229
I20260927 03:30:34.647924 14663 raft_consensus.cc:2229] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:30:34.649930 14663 raft_consensus.cc:2258] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:30:34.650339 14663 ts_tablet_manager.cc:1905] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:30:34.651702 14663 ts_tablet_manager.cc:1918] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:30:34.651783 14663 log.cc:1196] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/wal/wals/5755bc931afd44c78f949d350dac8f54
I20260927 03:30:34.652026 14663 ts_tablet_manager.cc:1939] T 5755bc931afd44c78f949d350dac8f54 P 1d48129deb7b4c02b81063b6d9ebc562: Deleting consensus metadata
I20260927 03:30:34.652423 14152 catalog_manager.cc:4841] TS 1d48129deb7b4c02b81063b6d9ebc562 (127.0.98.3:38037): tablet 5755bc931afd44c78f949d350dac8f54 (table test-table [id=8a9641e70dbe4e12ad8def716535965e]) successfully deleted
W20260927 03:30:34.655469 14154 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46393 (error 108)
I20260927 03:30:34.655660 14664 tablet_replica.cc:330] T 5755bc931afd44c78f949d350dac8f54 P 63bd9fc884d64ba6a55e09100051defc: stopping tablet replica
I20260927 03:30:34.655690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 14363
I20260927 03:30:34.656687 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:34.656553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46393 (local address 127.0.0.1:41776)
0927 03:30:34.656612 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:34.656646 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 03:30:34.656805 14154 catalog_manager.cc:4584] TS 1616528b7a99454abb6205c58f44d4e9 (127.0.98.1:46393): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5755bc931afd44c78f949d350dac8f54: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46393: connect: Connection refused (error 111)
I20260927 03:30:34.660949 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:34.660841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39885 (local address 127.0.0.1:35566)
0927 03:30:34.660880 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:34.660914 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260927 03:30:34.660952 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 14497
I20260927 03:30:34.665642 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 14135
I20260927 03:30:34.670871 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42247
--webserver_interface=127.0.98.62
--webserver_port=34017
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:30:34.751312 14667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:30:34.751610 14667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:30:34.751637 14667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:30:34.752792 14667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:30:34.752832 14667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:30:34.752846 14667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:30:34.752858 14667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:30:34.752871 14667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:30:34.754374 14667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:30:34.754617 14667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:30:34.754868 14667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:30:34.763594 14674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:34.763610 14673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:34.763613 14676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:34.763909 14667 server_base.cc:1034] running on GCE node
I20260927 03:30:34.764147 14667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:30:34.764423 14667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:30:34.765563 14667 hybrid_clock.cc:648] HybridClock initialized: now 1790479834765556 us; error 30 us; skew 500 ppm
Sep 27 03:30:34 dist-test-slave-j0hf krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:30:34.768081 14667 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:30:34.768581 14667 webserver.cc:460] Webserver started at http://127.0.98.62:34017/ using document root <none> and password file <none>
I20260927 03:30:34.768831 14667 fs_manager.cc:359] Metadata directory not provided
I20260927 03:30:34.768905 14667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:30:34.770303 14667 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:30:34.770960 14683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:34.771149 14667 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:30:34.771245 14667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "1eb88ac207a0431da5a2ab506e7c4439"
format_stamp: "Formatted at 2026-09-27 03:30:33 on dist-test-slave-j0hf"
server_key: "dab5835241f9ed84c17a1fc6f011a009"
server_key_iv: "3bbc06dab2bba2bb22de2cfab2bdf3db"
server_key_version: "encryptionkey@0"
I20260927 03:30:34.771732 14667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:30:34.797237 14667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:30:34.799708 14667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:30:34.799824 14667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:30:34.803901 14667 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42247
I20260927 03:30:34.803965 14735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42247 every 8 connection(s)
I20260927 03:30:34.804355 14667 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:30:34.805794 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 14667
I20260927 03:30:34.805999 14736 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:30:34.806128 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46393
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42247
--webserver_port=44629
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:30:34.806784 14736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439: Bootstrap starting.
I20260927 03:30:34.815091 14736 log.cc:824] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439: Log is configured to *not* fsync() on all Append() calls
I20260927 03:30:34.817008 14736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:30:34.817292 14736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439: Bootstrap complete.
I20260927 03:30:34.819618 14736 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb88ac207a0431da5a2ab506e7c4439" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42247 } }
I20260927 03:30:34.819900 14736 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb88ac207a0431da5a2ab506e7c4439, State: Initialized, Role: FOLLOWER
I20260927 03:30:34.820101 14736 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb88ac207a0431da5a2ab506e7c4439" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42247 } }
I20260927 03:30:34.820201 14736 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:30:34.820261 14736 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:30:34.820308 14736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [term 1 FOLLOWER]: Advancing to term 2
I20260927 03:30:34.821283 14736 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb88ac207a0431da5a2ab506e7c4439" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42247 } }
I20260927 03:30:34.821403 14736 leader_election.cc:302] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1eb88ac207a0431da5a2ab506e7c4439; no voters:
I20260927 03:30:34.821596 14736 leader_election.cc:288] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 03:30:34.821633 14740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 03:30:34.821743 14740 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [term 2 LEADER]: Becoming Leader. State: Replica: 1eb88ac207a0431da5a2ab506e7c4439, State: Running, Role: LEADER
I20260927 03:30:34.821871 14736 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:30:34.821890 14740 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb88ac207a0431da5a2ab506e7c4439" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42247 } }
I20260927 03:30:34.822264 14741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1eb88ac207a0431da5a2ab506e7c4439" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb88ac207a0431da5a2ab506e7c4439" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42247 } } }
I20260927 03:30:34.822345 14741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [sys.catalog]: This master's current role is: LEADER
I20260927 03:30:34.822557 14742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1eb88ac207a0431da5a2ab506e7c4439. Latest consensus state: current_term: 2 leader_uuid: "1eb88ac207a0431da5a2ab506e7c4439" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb88ac207a0431da5a2ab506e7c4439" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42247 } } }
I20260927 03:30:34.822638 14742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [sys.catalog]: This master's current role is: LEADER
I20260927 03:30:34.823746 14756 catalog_manager.cc:1259] Loaded cluster ID: b4222c1023a74777b1e31fd182888bb6
I20260927 03:30:34.823794 14756 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439: loading cluster ID for follower catalog manager: success
W20260927 03:30:34.823989 14756 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260927 03:30:34.824028 14756 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260927 03:30:34.824060 14746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:30:34.824388 14746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:30:34.824450 14746 catalog_manager.cc:1259] Loaded cluster ID: b4222c1023a74777b1e31fd182888bb6
I20260927 03:30:34.824471 14746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:30:34.824601 14746 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 03:30:34.829366 14746 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 03:30:34.832100 14746 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:30:34.832304 14746 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439: Loaded TSK: 0
I20260927 03:30:34.832530 14746 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 03:30:34.889712 14738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:30:34.889942 14738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:30:34.889982 14738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:30:34.891270 14738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:30:34.891336 14738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:30:34.891382 14738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:30:34.892836 14738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46393
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44629
--tserver_master_addrs=127.0.98.62:42247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:30:34.893116 14738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:30:34.893349 14738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:30:34.898347 14763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:34.898322 14766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:34.898377 14764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:34.898654 14738 server_base.cc:1034] running on GCE node
I20260927 03:30:34.898919 14738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:30:34.899189 14738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:30:34.900347 14738 hybrid_clock.cc:648] HybridClock initialized: now 1790479834900314 us; error 47 us; skew 500 ppm
Sep 27 03:30:34 dist-test-slave-j0hf krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:30:34.903126 14738 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:30:34.903716 14738 webserver.cc:460] Webserver started at http://127.0.98.1:44629/ using document root <none> and password file <none>
I20260927 03:30:34.903982 14738 fs_manager.cc:359] Metadata directory not provided
I20260927 03:30:34.904069 14738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:30:34.905838 14738 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:30:34.906595 14773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:34.906795 14738 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:30:34.906893 14738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "1616528b7a99454abb6205c58f44d4e9"
format_stamp: "Formatted at 2026-09-27 03:30:34 on dist-test-slave-j0hf"
server_key: "e9746c620ac559e974335a81cb02a692"
server_key_iv: "399eca5579795a64b969cf5c5bd745cf"
server_key_version: "encryptionkey@0"
I20260927 03:30:34.907327 14738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:30:34.928766 14738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:30:34.931511 14738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:30:34.931715 14738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:30:34.931969 14738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:30:34.932499 14780 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 03:30:34.934409 14738 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 03:30:34.934480 14738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 03:30:34.934543 14738 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 03:30:34.935217 14738 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 03:30:34.935277 14738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 03:30:34.935345 14780 tablet_bootstrap.cc:492] T 5755bc931afd44c78f949d350dac8f54 P 1616528b7a99454abb6205c58f44d4e9: Bootstrap starting.
I20260927 03:30:34.943256 14738 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46393
I20260927 03:30:34.943573 14887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46393 every 8 connection(s)
I20260927 03:30:34.943776 14738 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:30:34.946214 14780 log.cc:824] T 5755bc931afd44c78f949d350dac8f54 P 1616528b7a99454abb6205c58f44d4e9: Log is configured to *not* fsync() on all Append() calls
I20260927 03:30:34.948536 14780 tablet_bootstrap.cc:492] T 5755bc931afd44c78f949d350dac8f54 P 1616528b7a99454abb6205c58f44d4e9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:30:34.948845 14780 tablet_bootstrap.cc:492] T 5755bc931afd44c78f949d350dac8f54 P 1616528b7a99454abb6205c58f44d4e9: Bootstrap complete.
I20260927 03:30:34.949854 14780 ts_tablet_manager.cc:1397] T 5755bc931afd44c78f949d350dac8f54 P 1616528b7a99454abb6205c58f44d4e9: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.011s
I20260927 03:30:34.951329 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 14738
I20260927 03:30:34.951064 14780 raft_consensus.cc:348] T 5755bc931afd44c78f949d350dac8f54 P 1616528b7a99454abb6205c58f44d4e9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d48129deb7b4c02b81063b6d9ebc562" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38037 } } peers { permanent_uuid: "1616528b7a99454abb6205c58f44d4e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46393 } } peers { permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 } }
I20260927 03:30:34.951417 14780 raft_consensus.cc:729] T 5755bc931afd44c78f949d350dac8f54 P 1616528b7a99454abb6205c58f44d4e9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1616528b7a99454abb6205c58f44d4e9, State: Initialized, Role: FOLLOWER
I20260927 03:30:34.951654 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39885
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42247
--webserver_port=37079
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:30:34.951656 14780 consensus_queue.cc:260] T 5755bc931afd44c78f949d350dac8f54 P 1616528b7a99454abb6205c58f44d4e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d48129deb7b4c02b81063b6d9ebc562" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38037 } } peers { permanent_uuid: "1616528b7a99454abb6205c58f44d4e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46393 } } peers { permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 } }
I20260927 03:30:34.951954 14780 ts_tablet_manager.cc:1428] T 5755bc931afd44c78f949d350dac8f54 P 1616528b7a99454abb6205c58f44d4e9: Time spent starting tablet: real 0.002s user 0.002s sys 0.003s
Sep 27 03:30:34 dist-test-slave-j0hf krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:30:34.957700 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:34.944535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.1:53213)
0927 03:30:34.944828 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:34.944877 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:34.945353 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:34.945916 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:34.945921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:34.946220 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:30:34.946507 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:34.946515 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.948287 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.948288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:34.949521 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:34.949525 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.950427 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.950428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:34.950558 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:34.951730 (+ 1172us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:30:34.951741 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:30:34.953670 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:30:34.955983 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:34.955990 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:34.955992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:34.956241 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:34.956532 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:34.956533 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:34.956534 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:34.956654 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:34.957049 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:30:34.957055 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:30:34.957201 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 03:30:34.957315 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 03:30:34.957495 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":91,"threads_started":1}
I20260927 03:30:34.957729 14891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:34.944615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53213 (local address 127.0.98.62:42247)
0927 03:30:34.945528 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:34.945535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:34.945564 (+ 29us) server_negotiation.cc:407] Connection header received
0927 03:30:34.945670 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:34.945675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:34.945722 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:34.945862 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:34.946662 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.948125 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.949629 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.950218 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.950296 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:34.953814 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:34.953819 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:34.953819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:34.953831 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:34.955881 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:34.956347 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:34.956351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:34.956351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:34.956446 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:34.956727 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:34.956728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:34.956729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:34.956969 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:34.957067 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:34.957369 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 03:30:34.957495 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":856,"thread_start_us":86,"threads_started":1}
I20260927 03:30:34.958606 14888 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42247
I20260927 03:30:34.958724 14888 heartbeater.cc:464] Registering TS with master...
I20260927 03:30:34.958901 14888 heartbeater.cc:511] Master 127.0.98.62:42247 requested a full tablet report, sending...
I20260927 03:30:34.959898 14700 ts_manager.cc:194] Registered new tserver with Master: 1616528b7a99454abb6205c58f44d4e9 (127.0.98.1:46393)
I20260927 03:30:34.961097 14700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53213
I20260927 03:30:34.961980 14888 heartbeater.cc:503] Master 127.0.98.62:42247 was elected leader, sending a full tablet report...
I20260927 03:30:34.966929 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:34.960561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46393 (local address 127.0.0.1:41778)
0927 03:30:34.963645 (+ 3084us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:34.963674 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:34.963732 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:34.963983 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:34.963986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:34.963988 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:30:34.964070 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:34.964071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.964919 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.964920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:34.965788 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:34.965791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.966589 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.966590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:34.966673 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:34.966685 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 03:30:34.966804 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 03:30:34.966860 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3056,"thread_start_us":103,"threads_started":1}
I20260927 03:30:34.966928 14896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:34.961423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41778 (local address 127.0.98.1:46393)
0927 03:30:34.963830 (+ 2407us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:34.963833 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:34.963848 (+ 15us) server_negotiation.cc:407] Connection header received
0927 03:30:34.963876 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:34.963878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:34.963888 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:34.963965 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:30:34.964165 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.964834 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.965893 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:34.966501 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:34.966589 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:34.966643 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:34.966823 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 03:30:34.966860 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2382,"thread_start_us":85,"threads_started":1}
I20260927 03:30:34.967576 14822 tablet_service.cc:1514] Processing DeleteTablet for tablet 5755bc931afd44c78f949d350dac8f54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:30:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41778
I20260927 03:30:34.968034 14897 tablet_replica.cc:330] stopping tablet replica
I20260927 03:30:34.968160 14897 raft_consensus.cc:2229] T 5755bc931afd44c78f949d350dac8f54 P 1616528b7a99454abb6205c58f44d4e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:30:34.968228 14897 raft_consensus.cc:2258] T 5755bc931afd44c78f949d350dac8f54 P 1616528b7a99454abb6205c58f44d4e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:30:34.968519 14897 ts_tablet_manager.cc:1905] T 5755bc931afd44c78f949d350dac8f54 P 1616528b7a99454abb6205c58f44d4e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:30:34.970516 14897 ts_tablet_manager.cc:1918] T 5755bc931afd44c78f949d350dac8f54 P 1616528b7a99454abb6205c58f44d4e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:30:34.970603 14897 log.cc:1196] T 5755bc931afd44c78f949d350dac8f54 P 1616528b7a99454abb6205c58f44d4e9: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/wal/wals/5755bc931afd44c78f949d350dac8f54
I20260927 03:30:34.970872 14897 ts_tablet_manager.cc:1939] T 5755bc931afd44c78f949d350dac8f54 P 1616528b7a99454abb6205c58f44d4e9: Deleting consensus metadata
I20260927 03:30:34.971393 14687 catalog_manager.cc:4841] TS 1616528b7a99454abb6205c58f44d4e9 (127.0.98.1:46393): tablet 5755bc931afd44c78f949d350dac8f54 (table test-table [id=8a9641e70dbe4e12ad8def716535965e]) successfully deleted
W20260927 03:30:35.040972 14894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:30:35.041174 14894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:30:35.041198 14894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:30:35.042400 14894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:30:35.042444 14894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:30:35.042471 14894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:30:35.043926 14894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37079
--tserver_master_addrs=127.0.98.62:42247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:30:35.044178 14894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:30:35.044417 14894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:30:35.049283 14905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:35.049309 14903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:35.049463 14894 server_base.cc:1034] running on GCE node
W20260927 03:30:35.049309 14902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:35.049762 14894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:30:35.049954 14894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:30:35.051141 14894 hybrid_clock.cc:648] HybridClock initialized: now 1790479835051113 us; error 43 us; skew 500 ppm
Sep 27 03:30:35 dist-test-slave-j0hf krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:30:35.053872 14894 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:30:35.054409 14894 webserver.cc:460] Webserver started at http://127.0.98.2:37079/ using document root <none> and password file <none>
I20260927 03:30:35.054644 14894 fs_manager.cc:359] Metadata directory not provided
I20260927 03:30:35.054703 14894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:30:35.056202 14894 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:30:35.056850 14912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:35.057125 14894 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:30:35.057250 14894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "63bd9fc884d64ba6a55e09100051defc"
format_stamp: "Formatted at 2026-09-27 03:30:34 on dist-test-slave-j0hf"
server_key: "07f780196cf5073bfea103fbd09d8ee6"
server_key_iv: "0a1b79957bcffafaac6859c4313eb226"
server_key_version: "encryptionkey@0"
I20260927 03:30:35.057572 14894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:30:35.078054 14894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:30:35.080300 14894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:30:35.080446 14894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:30:35.080689 14894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:30:35.081095 14919 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 03:30:35.082695 14894 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 03:30:35.082753 14894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 03:30:35.082813 14894 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 03:30:35.083521 14894 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 03:30:35.083575 14894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260927 03:30:35.083657 14919 tablet_bootstrap.cc:492] T 5755bc931afd44c78f949d350dac8f54 P 63bd9fc884d64ba6a55e09100051defc: Bootstrap starting.
I20260927 03:30:35.090546 14894 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39885
I20260927 03:30:35.090638 15026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39885 every 8 connection(s)
I20260927 03:30:35.090974 14894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:30:35.093192 14919 log.cc:824] T 5755bc931afd44c78f949d350dac8f54 P 63bd9fc884d64ba6a55e09100051defc: Log is configured to *not* fsync() on all Append() calls
I20260927 03:30:35.095642 14919 tablet_bootstrap.cc:492] T 5755bc931afd44c78f949d350dac8f54 P 63bd9fc884d64ba6a55e09100051defc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:30:35.095932 14919 tablet_bootstrap.cc:492] T 5755bc931afd44c78f949d350dac8f54 P 63bd9fc884d64ba6a55e09100051defc: Bootstrap complete.
I20260927 03:30:35.097061 14919 ts_tablet_manager.cc:1397] T 5755bc931afd44c78f949d350dac8f54 P 63bd9fc884d64ba6a55e09100051defc: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.009s
Sep 27 03:30:35 dist-test-slave-j0hf krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:30:35.097616 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 14894
I20260927 03:30:35.097926 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38037
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42247
--webserver_port=38203
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:30:35.098523 14919 raft_consensus.cc:348] T 5755bc931afd44c78f949d350dac8f54 P 63bd9fc884d64ba6a55e09100051defc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d48129deb7b4c02b81063b6d9ebc562" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38037 } } peers { permanent_uuid: "1616528b7a99454abb6205c58f44d4e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46393 } } peers { permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 } }
I20260927 03:30:35.098917 14919 raft_consensus.cc:729] T 5755bc931afd44c78f949d350dac8f54 P 63bd9fc884d64ba6a55e09100051defc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63bd9fc884d64ba6a55e09100051defc, State: Initialized, Role: FOLLOWER
I20260927 03:30:35.099098 14919 consensus_queue.cc:260] T 5755bc931afd44c78f949d350dac8f54 P 63bd9fc884d64ba6a55e09100051defc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d48129deb7b4c02b81063b6d9ebc562" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38037 } } peers { permanent_uuid: "1616528b7a99454abb6205c58f44d4e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46393 } } peers { permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 } }
I20260927 03:30:35.099489 14919 ts_tablet_manager.cc:1428] T 5755bc931afd44c78f949d350dac8f54 P 63bd9fc884d64ba6a55e09100051defc: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260927 03:30:35.101151 14891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:35.091792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35125 (local address 127.0.98.62:42247)
0927 03:30:35.091850 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:35.091853 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:35.092085 (+ 232us) server_negotiation.cc:407] Connection header received
0927 03:30:35.092543 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:35.092545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:35.092562 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:35.092644 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:35.093231 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:35.093969 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:35.094971 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:35.095280 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:35.095359 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:35.097835 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:35.097837 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:35.097837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:35.097863 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:35.099656 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:35.100021 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:35.100022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:35.100023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:35.100085 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:35.100328 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:35.100329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:35.100329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:35.100468 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:35.100532 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:35.100949 (+ 417us) server_negotiation.cc:299] Negotiation successful
0927 03:30:35.101083 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 03:30:35.101333 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:35.091738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.2:35125)
0927 03:30:35.091925 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:35.091961 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:35.092428 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:35.092695 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:35.092699 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:35.092954 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:30:35.093132 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:35.093136 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:35.094087 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:35.094088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:35.094889 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:35.094891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:35.095381 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:35.095382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:35.095555 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:35.096104 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:30:35.096120 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:30:35.097735 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:30:35.099738 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:35.099742 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:35.099744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:35.099947 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:35.100151 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:35.100151 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:35.100151 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:35.100264 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:35.100564 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:30:35.100571 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:30:35.100748 (+ 177us) client_negotiation.cc:769] Sending connection context
0927 03:30:35.100896 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 03:30:35.101082 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":94,"threads_started":1}
I20260927 03:30:35.101854 15027 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42247
I20260927 03:30:35.101966 15027 heartbeater.cc:464] Registering TS with master...
I20260927 03:30:35.102167 15027 heartbeater.cc:511] Master 127.0.98.62:42247 requested a full tablet report, sending...
I20260927 03:30:35.102770 14700 ts_manager.cc:194] Registered new tserver with Master: 63bd9fc884d64ba6a55e09100051defc (127.0.98.2:39885)
I20260927 03:30:35.103483 14700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35125
I20260927 03:30:35.104422 15027 heartbeater.cc:503] Master 127.0.98.62:42247 was elected leader, sending a full tablet report...
Sep 27 03:30:35 dist-test-slave-j0hf krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 03:30:35.111742 15033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:35.103192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35570 (local address 127.0.98.2:39885)
0927 03:30:35.103344 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:35.103348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:35.103366 (+ 18us) server_negotiation.cc:407] Connection header received
0927 03:30:35.103395 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:35.103396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:35.103410 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:35.104138 (+ 728us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:35.104279 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:35.105174 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:35.106366 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:35.106647 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:35.106712 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:35.108935 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:35.108942 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:35.108942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:35.108950 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:35.110497 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:35.110901 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:35.110905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:35.110905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:35.110965 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:35.111170 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:35.111171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:35.111172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:35.111307 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:35.111403 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:35.111577 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 03:30:35.111649 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":98,"threads_started":1}
I20260927 03:30:35.111778 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:35.103133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39885 (local address 127.0.0.1:35570)
0927 03:30:35.103175 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:35.103186 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:35.103228 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:35.103635 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:35.103636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:35.104006 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:30:35.104056 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:35.104058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:35.105252 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:35.105253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:35.106259 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:35.106262 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:35.106731 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:35.106734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:35.106804 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:35.107305 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:30:35.107311 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:30:35.108844 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:30:35.110604 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:35.110612 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:35.110614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:35.110816 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:35.111034 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:35.111035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:35.111036 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:35.111096 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:35.111382 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:30:35.111389 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:30:35.111465 (+ 76us) client_negotiation.cc:769] Sending connection context
0927 03:30:35.111537 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 03:30:35.111648 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260927 03:30:35.112267 14961 tablet_service.cc:1514] Processing DeleteTablet for tablet 5755bc931afd44c78f949d350dac8f54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:30:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35570
I20260927 03:30:35.112764 15034 tablet_replica.cc:330] stopping tablet replica
I20260927 03:30:35.112875 15034 raft_consensus.cc:2229] T 5755bc931afd44c78f949d350dac8f54 P 63bd9fc884d64ba6a55e09100051defc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:30:35.112949 15034 raft_consensus.cc:2258] T 5755bc931afd44c78f949d350dac8f54 P 63bd9fc884d64ba6a55e09100051defc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:30:35.113248 15034 ts_tablet_manager.cc:1905] T 5755bc931afd44c78f949d350dac8f54 P 63bd9fc884d64ba6a55e09100051defc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:30:35.115465 15034 ts_tablet_manager.cc:1918] T 5755bc931afd44c78f949d350dac8f54 P 63bd9fc884d64ba6a55e09100051defc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:30:35.115545 15034 log.cc:1196] T 5755bc931afd44c78f949d350dac8f54 P 63bd9fc884d64ba6a55e09100051defc: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/wal/wals/5755bc931afd44c78f949d350dac8f54
I20260927 03:30:35.115784 15034 ts_tablet_manager.cc:1939] T 5755bc931afd44c78f949d350dac8f54 P 63bd9fc884d64ba6a55e09100051defc: Deleting consensus metadata
I20260927 03:30:35.116282 14685 catalog_manager.cc:4841] TS 63bd9fc884d64ba6a55e09100051defc (127.0.98.2:39885): tablet 5755bc931afd44c78f949d350dac8f54 (table test-table [id=8a9641e70dbe4e12ad8def716535965e]) successfully deleted
W20260927 03:30:35.183471 15031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:30:35.183696 15031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:30:35.183729 15031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:30:35.184898 15031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:30:35.184966 15031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:30:35.185009 15031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:30:35.186453 15031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38203
--tserver_master_addrs=127.0.98.62:42247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:30:35.186707 15031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:30:35.186940 15031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:30:35.191804 15040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:35.191838 15039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:35.191804 15042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:35.191987 15031 server_base.cc:1034] running on GCE node
I20260927 03:30:35.192207 15031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:30:35.192389 15031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:30:35.193535 15031 hybrid_clock.cc:648] HybridClock initialized: now 1790479835193517 us; error 27 us; skew 500 ppm
Sep 27 03:30:35 dist-test-slave-j0hf krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:30:35.196194 15031 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:30:35.196601 15031 webserver.cc:460] Webserver started at http://127.0.98.3:38203/ using document root <none> and password file <none>
I20260927 03:30:35.196821 15031 fs_manager.cc:359] Metadata directory not provided
I20260927 03:30:35.196902 15031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:30:35.198192 15031 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:30:35.198745 15049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:35.198910 15031 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:30:35.199009 15031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "1d48129deb7b4c02b81063b6d9ebc562"
format_stamp: "Formatted at 2026-09-27 03:30:34 on dist-test-slave-j0hf"
server_key: "d466fd0bfc3ea9b165c5dfcad1a18e93"
server_key_iv: "ac2decac81bd1c7a4ed079e0ca8ffb83"
server_key_version: "encryptionkey@0"
I20260927 03:30:35.199378 15031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:30:35.212603 15031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:30:35.214377 15031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:30:35.214532 15031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:30:35.214742 15031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:30:35.215029 15031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:30:35.215091 15031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:35.215131 15031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:30:35.215159 15031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:35.220902 15031 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38037
I20260927 03:30:35.220971 15162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38037 every 8 connection(s)
I20260927 03:30:35.221335 15031 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:30:35.223733 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 15031
Sep 27 03:30:35 dist-test-slave-j0hf krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:30:35.231308 15167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:35.224121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54164 (local address 127.0.98.62:42247)
0927 03:30:35.224244 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:35.224245 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:35.224892 (+ 647us) server_negotiation.cc:407] Connection header received
0927 03:30:35.224947 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:35.224948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:35.224957 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:35.225010 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:35.225654 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:35.226099 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:35.226956 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:35.227157 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:35.227232 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:35.228423 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:35.228425 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:35.228425 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:35.228431 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:35.230072 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:35.230487 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:35.230488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:35.230488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:35.230543 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:35.230716 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:35.230717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:35.230718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:35.230861 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:35.231041 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:35.231079 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 03:30:35.231168 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":81,"threads_started":1}
I20260927 03:30:35.231945 14891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:35.222004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55749 (local address 127.0.98.62:42247)
0927 03:30:35.222057 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:35.222059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:35.222312 (+ 253us) server_negotiation.cc:407] Connection header received
0927 03:30:35.222839 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:35.222840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:35.222849 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:35.222901 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:35.224133 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:35.224677 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:35.226004 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:35.226206 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:35.226280 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:35.228514 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:35.228515 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:35.228515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:35.228521 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:35.230331 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:35.230833 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:35.230834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:35.230834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:35.230878 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:35.231168 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:35.231169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:35.231169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:35.231293 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:35.231357 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:35.231787 (+ 430us) server_negotiation.cc:299] Negotiation successful
0927 03:30:35.231866 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:30:35.232139 15165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:35.222021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.3:55749)
0927 03:30:35.222190 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:35.222231 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:35.222753 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:35.223000 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:35.223003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:35.223671 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:30:35.223953 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:35.223958 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:35.224773 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:35.224774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:35.225922 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:35.225924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:35.226285 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:35.226286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:35.226409 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:35.226898 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:30:35.226924 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:30:35.228421 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:30:35.230415 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:35.230421 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:35.230423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:35.230763 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:35.230963 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:35.230963 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:35.230964 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:35.231090 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:35.231357 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:30:35.231364 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:30:35.231635 (+ 271us) client_negotiation.cc:769] Sending connection context
0927 03:30:35.231754 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 03:30:35.231901 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
I20260927 03:30:35.232501 15163 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42247
I20260927 03:30:35.232594 15163 heartbeater.cc:464] Registering TS with master...
I20260927 03:30:35.232769 15163 heartbeater.cc:511] Master 127.0.98.62:42247 requested a full tablet report, sending...
I20260927 03:30:35.233111 14700 ts_manager.cc:194] Registered new tserver with Master: 1d48129deb7b4c02b81063b6d9ebc562 (127.0.98.3:38037)
I20260927 03:30:35.233460 14700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55749
I20260927 03:30:35.243167 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:30:35.245903 14891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:35.243605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54172 (local address 127.0.98.62:42247)
0927 03:30:35.243668 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:35.243670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:35.243746 (+ 76us) server_negotiation.cc:407] Connection header received
0927 03:30:35.243815 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:35.243816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:35.243848 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:35.243906 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:30:35.244005 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:35.244467 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:35.245239 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:35.245407 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:35.245454 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:35.245590 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:30:35.245687 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:30:35.245748 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:35.245818 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 03:30:35.245833 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:30:35.246222 14700 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54172:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:30:35.247828 14700 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:30:35.249934 14822 tablet_service.cc:1467] Processing CreateTablet for tablet 2715d6ed3ba64bc4bc1b0875ccf0aecd (DEFAULT_TABLE table=test-table [id=e484a287e1134faf9b94d8d5db936f98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:30:35.249976 14961 tablet_service.cc:1467] Processing CreateTablet for tablet 2715d6ed3ba64bc4bc1b0875ccf0aecd (DEFAULT_TABLE table=test-table [id=e484a287e1134faf9b94d8d5db936f98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:30:35.250149 14961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2715d6ed3ba64bc4bc1b0875ccf0aecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:30:35.250149 14822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2715d6ed3ba64bc4bc1b0875ccf0aecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:30:35.251540 14780 tablet_bootstrap.cc:492] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9: Bootstrap starting.
I20260927 03:30:35.251782 14919 tablet_bootstrap.cc:492] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 63bd9fc884d64ba6a55e09100051defc: Bootstrap starting.
I20260927 03:30:35.251859 14780 tablet_bootstrap.cc:654] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9: Neither blocks nor log segments found. Creating new log.
I20260927 03:30:35.252115 14919 tablet_bootstrap.cc:654] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 63bd9fc884d64ba6a55e09100051defc: Neither blocks nor log segments found. Creating new log.
I20260927 03:30:35.252470 14780 tablet_bootstrap.cc:492] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9: No bootstrap required, opened a new log
I20260927 03:30:35.252535 14780 ts_tablet_manager.cc:1397] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 03:30:35.252650 14780 raft_consensus.cc:348] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d48129deb7b4c02b81063b6d9ebc562" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38037 } } peers { permanent_uuid: "1616528b7a99454abb6205c58f44d4e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46393 } } peers { permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 } }
I20260927 03:30:35.252719 14780 raft_consensus.cc:374] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:30:35.252767 14780 raft_consensus.cc:729] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1616528b7a99454abb6205c58f44d4e9, State: Initialized, Role: FOLLOWER
I20260927 03:30:35.252830 14780 consensus_queue.cc:260] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d48129deb7b4c02b81063b6d9ebc562" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38037 } } peers { permanent_uuid: "1616528b7a99454abb6205c58f44d4e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46393 } } peers { permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 } }
I20260927 03:30:35.252933 14780 ts_tablet_manager.cc:1428] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:35.253430 14919 tablet_bootstrap.cc:492] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 63bd9fc884d64ba6a55e09100051defc: No bootstrap required, opened a new log
I20260927 03:30:35.253484 14919 ts_tablet_manager.cc:1397] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 63bd9fc884d64ba6a55e09100051defc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:30:35.253652 14919 raft_consensus.cc:348] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 63bd9fc884d64ba6a55e09100051defc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d48129deb7b4c02b81063b6d9ebc562" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38037 } } peers { permanent_uuid: "1616528b7a99454abb6205c58f44d4e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46393 } } peers { permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 } }
I20260927 03:30:35.253720 14919 raft_consensus.cc:374] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 63bd9fc884d64ba6a55e09100051defc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:30:35.253775 14919 raft_consensus.cc:729] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 63bd9fc884d64ba6a55e09100051defc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63bd9fc884d64ba6a55e09100051defc, State: Initialized, Role: FOLLOWER
I20260927 03:30:35.253826 14919 consensus_queue.cc:260] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 63bd9fc884d64ba6a55e09100051defc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d48129deb7b4c02b81063b6d9ebc562" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38037 } } peers { permanent_uuid: "1616528b7a99454abb6205c58f44d4e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46393 } } peers { permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 } }
I20260927 03:30:35.253912 14919 ts_tablet_manager.cc:1428] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 63bd9fc884d64ba6a55e09100051defc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:35.255343 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:35.249711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38037 (local address 127.0.0.1:45974)
0927 03:30:35.249745 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:35.249755 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:35.249789 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:35.251948 (+ 2159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:35.251949 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:35.251951 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:30:35.252002 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:35.252004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:35.253308 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:35.253309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:35.254332 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:35.254336 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:35.255112 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:35.255113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:35.255176 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:35.255181 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:30:35.255234 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 03:30:35.255275 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260927 03:30:35.255357 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:35.251587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45974 (local address 127.0.98.3:38037)
0927 03:30:35.251765 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:35.251771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:35.251781 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:30:35.251820 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:35.251822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:35.251838 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:35.251929 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:30:35.252097 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:35.253186 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:35.254416 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:35.255024 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:35.255114 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:35.255174 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:35.255245 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 03:30:35.255280 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":105,"threads_started":1}
I20260927 03:30:35.256042 15097 tablet_service.cc:1467] Processing CreateTablet for tablet 2715d6ed3ba64bc4bc1b0875ccf0aecd (DEFAULT_TABLE table=test-table [id=e484a287e1134faf9b94d8d5db936f98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:30:35.256356 15097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2715d6ed3ba64bc4bc1b0875ccf0aecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:30:35.258852 15172 tablet_bootstrap.cc:492] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1d48129deb7b4c02b81063b6d9ebc562: Bootstrap starting.
I20260927 03:30:35.259493 15172 tablet_bootstrap.cc:654] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1d48129deb7b4c02b81063b6d9ebc562: Neither blocks nor log segments found. Creating new log.
I20260927 03:30:35.259977 15172 log.cc:824] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1d48129deb7b4c02b81063b6d9ebc562: Log is configured to *not* fsync() on all Append() calls
I20260927 03:30:35.260684 15172 tablet_bootstrap.cc:492] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1d48129deb7b4c02b81063b6d9ebc562: No bootstrap required, opened a new log
I20260927 03:30:35.260737 15172 ts_tablet_manager.cc:1397] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1d48129deb7b4c02b81063b6d9ebc562: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:30:35.262274 15172 raft_consensus.cc:348] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1d48129deb7b4c02b81063b6d9ebc562 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d48129deb7b4c02b81063b6d9ebc562" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38037 } } peers { permanent_uuid: "1616528b7a99454abb6205c58f44d4e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46393 } } peers { permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 } }
I20260927 03:30:35.262406 15172 raft_consensus.cc:374] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1d48129deb7b4c02b81063b6d9ebc562 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:30:35.262427 15172 raft_consensus.cc:729] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1d48129deb7b4c02b81063b6d9ebc562 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d48129deb7b4c02b81063b6d9ebc562, State: Initialized, Role: FOLLOWER
I20260927 03:30:35.262511 15172 consensus_queue.cc:260] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1d48129deb7b4c02b81063b6d9ebc562 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d48129deb7b4c02b81063b6d9ebc562" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38037 } } peers { permanent_uuid: "1616528b7a99454abb6205c58f44d4e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46393 } } peers { permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 } }
I20260927 03:30:35.262770 15172 ts_tablet_manager.cc:1428] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1d48129deb7b4c02b81063b6d9ebc562: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 03:30:35.262789 15163 heartbeater.cc:503] Master 127.0.98.62:42247 was elected leader, sending a full tablet report...
W20260927 03:30:35.341706 15028 tablet.cc:2369] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 63bd9fc884d64ba6a55e09100051defc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:30:35.379319 14893 raft_consensus.cc:482] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:30:35.379509 14893 raft_consensus.cc:504] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d48129deb7b4c02b81063b6d9ebc562" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38037 } } peers { permanent_uuid: "1616528b7a99454abb6205c58f44d4e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46393 } } peers { permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 } }
I20260927 03:30:35.380355 14893 leader_election.cc:288] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d48129deb7b4c02b81063b6d9ebc562 (127.0.98.3:38037), 63bd9fc884d64ba6a55e09100051defc (127.0.98.2:39885)
I20260927 03:30:35.383524 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:35.380657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47285 (local address 127.0.98.3:38037)
0927 03:30:35.380726 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:35.380728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:35.380739 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:30:35.380770 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:35.380771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:35.380783 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:35.380895 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:30:35.381140 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:35.381699 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:35.382735 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:35.383128 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:35.383183 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:35.383190 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:35.383376 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 03:30:35.383454 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 03:30:35.383524 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:35.380559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38037 (local address 127.0.98.1:47285)
0927 03:30:35.380620 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:35.380633 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:35.380681 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:35.380944 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:35.380945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:35.380947 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:30:35.381028 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:35.381030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:35.381783 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:35.381784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:35.382633 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:35.382636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:35.383214 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:35.383215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:35.383296 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:35.383300 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:30:35.383352 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 03:30:35.383396 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":52}
I20260927 03:30:35.383602 15033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:35.380775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49399 (local address 127.0.98.2:39885)
0927 03:30:35.380852 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:35.380854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:35.380957 (+ 103us) server_negotiation.cc:407] Connection header received
0927 03:30:35.381016 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:35.381017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:35.381044 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:35.381114 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:30:35.381289 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:35.381775 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:35.382776 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:35.383193 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:35.383266 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:35.383343 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:35.383494 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 03:30:35.383534 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 03:30:35.383659 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:35.380700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39885 (local address 127.0.98.1:49399)
0927 03:30:35.380882 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:35.380893 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:35.380951 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:35.381135 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:35.381136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:35.381138 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:30:35.381217 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:35.381219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:35.381864 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:35.381864 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:35.382649 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:35.382651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:35.383296 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:35.383297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:35.383376 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:35.383381 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:30:35.383484 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 03:30:35.383522 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":123,"threads_started":1}
I20260927 03:30:35.383875 15117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2715d6ed3ba64bc4bc1b0875ccf0aecd" candidate_uuid: "1616528b7a99454abb6205c58f44d4e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d48129deb7b4c02b81063b6d9ebc562" is_pre_election: true
I20260927 03:30:35.383975 14981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2715d6ed3ba64bc4bc1b0875ccf0aecd" candidate_uuid: "1616528b7a99454abb6205c58f44d4e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63bd9fc884d64ba6a55e09100051defc" is_pre_election: true
I20260927 03:30:35.384025 15117 raft_consensus.cc:2454] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1d48129deb7b4c02b81063b6d9ebc562 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1616528b7a99454abb6205c58f44d4e9 in term 0.
I20260927 03:30:35.384155 14981 raft_consensus.cc:2454] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 63bd9fc884d64ba6a55e09100051defc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1616528b7a99454abb6205c58f44d4e9 in term 0.
I20260927 03:30:35.384327 14775 leader_election.cc:302] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1616528b7a99454abb6205c58f44d4e9, 1d48129deb7b4c02b81063b6d9ebc562; no voters:
I20260927 03:30:35.384457 14893 raft_consensus.cc:2781] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:30:35.384512 14893 raft_consensus.cc:482] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:30:35.384531 14893 raft_consensus.cc:3037] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:30:35.385195 14893 raft_consensus.cc:504] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d48129deb7b4c02b81063b6d9ebc562" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38037 } } peers { permanent_uuid: "1616528b7a99454abb6205c58f44d4e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46393 } } peers { permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 } }
I20260927 03:30:35.385329 14893 leader_election.cc:288] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [CANDIDATE]: Term 1 election: Requested vote from peers 1d48129deb7b4c02b81063b6d9ebc562 (127.0.98.3:38037), 63bd9fc884d64ba6a55e09100051defc (127.0.98.2:39885)
I20260927 03:30:35.385515 14981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2715d6ed3ba64bc4bc1b0875ccf0aecd" candidate_uuid: "1616528b7a99454abb6205c58f44d4e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63bd9fc884d64ba6a55e09100051defc"
I20260927 03:30:35.385535 15117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2715d6ed3ba64bc4bc1b0875ccf0aecd" candidate_uuid: "1616528b7a99454abb6205c58f44d4e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d48129deb7b4c02b81063b6d9ebc562"
I20260927 03:30:35.385607 14981 raft_consensus.cc:3037] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 63bd9fc884d64ba6a55e09100051defc [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:30:35.385612 15117 raft_consensus.cc:3037] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1d48129deb7b4c02b81063b6d9ebc562 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:30:35.386492 15117 raft_consensus.cc:2454] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1d48129deb7b4c02b81063b6d9ebc562 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1616528b7a99454abb6205c58f44d4e9 in term 1.
I20260927 03:30:35.386492 14981 raft_consensus.cc:2454] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 63bd9fc884d64ba6a55e09100051defc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1616528b7a99454abb6205c58f44d4e9 in term 1.
I20260927 03:30:35.386658 14775 leader_election.cc:302] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1616528b7a99454abb6205c58f44d4e9, 1d48129deb7b4c02b81063b6d9ebc562; no voters:
I20260927 03:30:35.386790 14893 raft_consensus.cc:2781] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:30:35.386960 14893 raft_consensus.cc:686] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [term 1 LEADER]: Becoming Leader. State: Replica: 1616528b7a99454abb6205c58f44d4e9, State: Running, Role: LEADER
I20260927 03:30:35.387019 14893 consensus_queue.cc:237] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d48129deb7b4c02b81063b6d9ebc562" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38037 } } peers { permanent_uuid: "1616528b7a99454abb6205c58f44d4e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46393 } } peers { permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 } }
I20260927 03:30:35.387866 14700 catalog_manager.cc:5495] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1616528b7a99454abb6205c58f44d4e9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1616528b7a99454abb6205c58f44d4e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d48129deb7b4c02b81063b6d9ebc562" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1616528b7a99454abb6205c58f44d4e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 } health_report { overall_health: UNKNOWN } } }
I20260927 03:30:35.398283 14896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:35.396225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41786 (local address 127.0.98.1:46393)
0927 03:30:35.396265 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:35.396267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:35.396275 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:30:35.396355 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:35.396356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:35.396373 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:35.396431 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:30:35.396528 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:35.396993 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:35.397661 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:35.397836 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:35.397886 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:35.397999 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:30:35.398092 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:30:35.398142 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:35.398177 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 03:30:35.398192 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:30:35.399889 15117 raft_consensus.cc:1261] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1d48129deb7b4c02b81063b6d9ebc562 [term 1 FOLLOWER]: Refusing update from remote peer 1616528b7a99454abb6205c58f44d4e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:30:35.399899 14981 raft_consensus.cc:1261] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 63bd9fc884d64ba6a55e09100051defc [term 1 FOLLOWER]: Refusing update from remote peer 1616528b7a99454abb6205c58f44d4e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:30:35.400261 15176 consensus_queue.cc:1035] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d48129deb7b4c02b81063b6d9ebc562" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:30:35.400395 14893 consensus_queue.cc:1035] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:30:35.402809 15182 mvcc.cc:205] Tried to move back new op lower bound from 7333805405794918400 to 7333805405745836032. Current Snapshot: MvccSnapshot[applied={T|T < 7333805405794918400}]
I20260927 03:30:35.402985 15180 mvcc.cc:205] Tried to move back new op lower bound from 7333805405794918400 to 7333805405745836032. Current Snapshot: MvccSnapshot[applied={T|T < 7333805405794918400}]
I20260927 03:30:35.405282 14700 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:30:35.405385 14700 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:30:35.406098 14700 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439: Sending DeleteTablet for 3 replicas of tablet 2715d6ed3ba64bc4bc1b0875ccf0aecd
I20260927 03:30:35.406455 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 14738
I20260927 03:30:35.406463 14822 tablet_service.cc:1514] Processing DeleteTablet for tablet 2715d6ed3ba64bc4bc1b0875ccf0aecd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:30:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41778
I20260927 03:30:35.406457 15097 tablet_service.cc:1514] Processing DeleteTablet for tablet 2715d6ed3ba64bc4bc1b0875ccf0aecd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:30:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45974
I20260927 03:30:35.406635 14961 tablet_service.cc:1514] Processing DeleteTablet for tablet 2715d6ed3ba64bc4bc1b0875ccf0aecd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:30:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35570
I20260927 03:30:35.407069 15034 tablet_replica.cc:330] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 63bd9fc884d64ba6a55e09100051defc: stopping tablet replica
I20260927 03:30:35.407122 15034 raft_consensus.cc:2229] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 63bd9fc884d64ba6a55e09100051defc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:30:35.407186 15034 raft_consensus.cc:2258] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 63bd9fc884d64ba6a55e09100051defc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:30:35.407575 15034 ts_tablet_manager.cc:1905] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 63bd9fc884d64ba6a55e09100051defc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:30:35.407622 15189 tablet_replica.cc:330] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1d48129deb7b4c02b81063b6d9ebc562: stopping tablet replica
I20260927 03:30:35.407724 15189 raft_consensus.cc:2229] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1d48129deb7b4c02b81063b6d9ebc562 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:30:35.407828 15189 raft_consensus.cc:2258] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1d48129deb7b4c02b81063b6d9ebc562 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:30:35.408442 15189 ts_tablet_manager.cc:1905] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1d48129deb7b4c02b81063b6d9ebc562: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:30:35.408977 15034 ts_tablet_manager.cc:1918] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 63bd9fc884d64ba6a55e09100051defc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:30:35.409051 15034 log.cc:1196] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 63bd9fc884d64ba6a55e09100051defc: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/wal/wals/2715d6ed3ba64bc4bc1b0875ccf0aecd
I20260927 03:30:35.409252 15034 ts_tablet_manager.cc:1939] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 63bd9fc884d64ba6a55e09100051defc: Deleting consensus metadata
I20260927 03:30:35.409560 14685 catalog_manager.cc:4841] TS 63bd9fc884d64ba6a55e09100051defc (127.0.98.2:39885): tablet 2715d6ed3ba64bc4bc1b0875ccf0aecd (table test-table [id=e484a287e1134faf9b94d8d5db936f98]) successfully deleted
I20260927 03:30:35.410470 15189 ts_tablet_manager.cc:1918] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1d48129deb7b4c02b81063b6d9ebc562: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:30:35.410547 15189 log.cc:1196] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1d48129deb7b4c02b81063b6d9ebc562: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/wal/wals/2715d6ed3ba64bc4bc1b0875ccf0aecd
I20260927 03:30:35.410818 15189 ts_tablet_manager.cc:1939] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1d48129deb7b4c02b81063b6d9ebc562: Deleting consensus metadata
I20260927 03:30:35.411289 14685 catalog_manager.cc:4841] TS 1d48129deb7b4c02b81063b6d9ebc562 (127.0.98.3:38037): tablet 2715d6ed3ba64bc4bc1b0875ccf0aecd (table test-table [id=e484a287e1134faf9b94d8d5db936f98]) successfully deleted
W20260927 03:30:35.413189 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46393 (error 108)
I20260927 03:30:35.413684 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:35.413565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46393 (local address 127.0.0.1:41796)
0927 03:30:35.413618 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:35.413659 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 03:30:35.413864 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 14894
W20260927 03:30:35.413820 14687 catalog_manager.cc:4584] TS 1616528b7a99454abb6205c58f44d4e9 (127.0.98.1:46393): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2715d6ed3ba64bc4bc1b0875ccf0aecd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46393: connect: Connection refused (error 111)
I20260927 03:30:35.419191 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 15031
I20260927 03:30:35.423692 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 14667
I20260927 03:30:35.429207 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42247
--webserver_interface=127.0.98.62
--webserver_port=34017
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42247
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:30:35.510473 15191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:30:35.510726 15191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:30:35.510751 15191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:30:35.511997 15191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:30:35.512040 15191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:30:35.512055 15191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:30:35.512068 15191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:30:35.512079 15191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:30:35.513521 15191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:30:35.513723 15191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:30:35.513942 15191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:30:35.518731 15196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:35.518780 15199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:35.518942 15191 server_base.cc:1034] running on GCE node
W20260927 03:30:35.518793 15197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:35.519284 15191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:30:35.519603 15191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:30:35.520774 15191 hybrid_clock.cc:648] HybridClock initialized: now 1790479835520760 us; error 30 us; skew 500 ppm
Sep 27 03:30:35 dist-test-slave-j0hf krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:30:35.523285 15191 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:30:35.523780 15191 webserver.cc:460] Webserver started at http://127.0.98.62:34017/ using document root <none> and password file <none>
I20260927 03:30:35.524017 15191 fs_manager.cc:359] Metadata directory not provided
I20260927 03:30:35.524062 15191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:30:35.525377 15191 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:30:35.526080 15206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:35.526265 15191 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:30:35.526361 15191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "1eb88ac207a0431da5a2ab506e7c4439"
format_stamp: "Formatted at 2026-09-27 03:30:33 on dist-test-slave-j0hf"
server_key: "dab5835241f9ed84c17a1fc6f011a009"
server_key_iv: "3bbc06dab2bba2bb22de2cfab2bdf3db"
server_key_version: "encryptionkey@0"
I20260927 03:30:35.526714 15191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:30:35.536422 15191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:30:35.538646 15191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:30:35.538772 15191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:30:35.542761 15191 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42247
I20260927 03:30:35.542822 15258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42247 every 8 connection(s)
I20260927 03:30:35.543219 15191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:30:35.543941 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 15191
I20260927 03:30:35.544247 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46393
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42247
--webserver_port=44629
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:30:35.545068 15259 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:30:35.545751 15259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439: Bootstrap starting.
I20260927 03:30:35.554373 15259 log.cc:824] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439: Log is configured to *not* fsync() on all Append() calls
I20260927 03:30:35.556886 15259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:30:35.557226 15259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439: Bootstrap complete.
I20260927 03:30:35.559664 15259 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb88ac207a0431da5a2ab506e7c4439" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42247 } }
I20260927 03:30:35.559934 15259 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb88ac207a0431da5a2ab506e7c4439, State: Initialized, Role: FOLLOWER
I20260927 03:30:35.560146 15259 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb88ac207a0431da5a2ab506e7c4439" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42247 } }
I20260927 03:30:35.560246 15259 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:30:35.560302 15259 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:30:35.560352 15259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [term 2 FOLLOWER]: Advancing to term 3
I20260927 03:30:35.561367 15259 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb88ac207a0431da5a2ab506e7c4439" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42247 } }
I20260927 03:30:35.561491 15259 leader_election.cc:302] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1eb88ac207a0431da5a2ab506e7c4439; no voters:
I20260927 03:30:35.561724 15259 leader_election.cc:288] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 03:30:35.561761 15263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 03:30:35.561878 15263 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [term 3 LEADER]: Becoming Leader. State: Replica: 1eb88ac207a0431da5a2ab506e7c4439, State: Running, Role: LEADER
I20260927 03:30:35.561966 15263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb88ac207a0431da5a2ab506e7c4439" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42247 } }
I20260927 03:30:35.562003 15259 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:30:35.562234 15264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1eb88ac207a0431da5a2ab506e7c4439" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb88ac207a0431da5a2ab506e7c4439" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42247 } } }
I20260927 03:30:35.562322 15264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [sys.catalog]: This master's current role is: LEADER
I20260927 03:30:35.562228 15263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1eb88ac207a0431da5a2ab506e7c4439. Latest consensus state: current_term: 3 leader_uuid: "1eb88ac207a0431da5a2ab506e7c4439" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb88ac207a0431da5a2ab506e7c4439" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42247 } } }
I20260927 03:30:35.562515 15263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eb88ac207a0431da5a2ab506e7c4439 [sys.catalog]: This master's current role is: LEADER
I20260927 03:30:35.562635 15268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:30:35.563794 15268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:30:35.563916 15268 catalog_manager.cc:1259] Loaded cluster ID: b4222c1023a74777b1e31fd182888bb6
I20260927 03:30:35.563992 15268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 03:30:35.564397 15268 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f5b1b8d79ad google::LogMessage::Fail() at ??:0
@ 0x7f5b1b8da88a google::LogMessage::SendToLog() at ??:0
@ 0x7f5b1b8d74de google::LogMessage::Flush() at ??:0
@ 0x7f5b1b8d8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5b237f80f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f5b238018b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f5b1d7a50d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5b1d79a954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5b1e5fe6db start_thread at ??:0
@ 0x7f5b1a3d561f clone at ??:0
W20260927 03:30:35.641121 15261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:30:35.641346 15261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:30:35.641384 15261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:30:35.642513 15261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:30:35.642578 15261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:30:35.642622 15261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:30:35.644165 15261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46393
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44629
--tserver_master_addrs=127.0.98.62:42247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:30:35.644428 15261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:30:35.644655 15261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:30:35.649468 15295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:35.649560 15261 server_base.cc:1034] running on GCE node
W20260927 03:30:35.649637 15296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:35.649457 15298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:35.649986 15261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:30:35.650231 15261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:30:35.651392 15261 hybrid_clock.cc:648] HybridClock initialized: now 1790479835651377 us; error 33 us; skew 500 ppm
Sep 27 03:30:35 dist-test-slave-j0hf krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:30:35.653851 15261 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:30:35.654284 15261 webserver.cc:460] Webserver started at http://127.0.98.1:44629/ using document root <none> and password file <none>
I20260927 03:30:35.654515 15261 fs_manager.cc:359] Metadata directory not provided
I20260927 03:30:35.654582 15261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:30:35.656014 15261 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:30:35.656746 15305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:35.656912 15261 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:30:35.657008 15261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "1616528b7a99454abb6205c58f44d4e9"
format_stamp: "Formatted at 2026-09-27 03:30:34 on dist-test-slave-j0hf"
server_key: "e9746c620ac559e974335a81cb02a692"
server_key_iv: "399eca5579795a64b969cf5c5bd745cf"
server_key_version: "encryptionkey@0"
I20260927 03:30:35.657372 15261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:30:35.680927 15261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:30:35.683167 15261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:30:35.683341 15261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:30:35.683593 15261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:30:35.684032 15312 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 03:30:35.686017 15261 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 03:30:35.686085 15261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 03:30:35.686164 15261 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 03:30:35.686774 15261 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 03:30:35.686847 15261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 03:30:35.686924 15312 tablet_bootstrap.cc:492] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9: Bootstrap starting.
I20260927 03:30:35.694013 15261 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46393
I20260927 03:30:35.694057 15419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46393 every 8 connection(s)
I20260927 03:30:35.694422 15261 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:30:35.697938 15312 log.cc:824] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9: Log is configured to *not* fsync() on all Append() calls
I20260927 03:30:35.699693 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 15261
I20260927 03:30:35.699994 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39885
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42247
--webserver_port=37079
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:30:35.700791 15312 tablet_bootstrap.cc:492] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:30:35.701089 15312 tablet_bootstrap.cc:492] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9: Bootstrap complete.
I20260927 03:30:35.702301 15312 ts_tablet_manager.cc:1397] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.009s
I20260927 03:30:35.703771 15312 raft_consensus.cc:348] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d48129deb7b4c02b81063b6d9ebc562" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38037 } } peers { permanent_uuid: "1616528b7a99454abb6205c58f44d4e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46393 } } peers { permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 } }
I20260927 03:30:35.704109 15312 raft_consensus.cc:729] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1616528b7a99454abb6205c58f44d4e9, State: Initialized, Role: FOLLOWER
I20260927 03:30:35.704275 15312 consensus_queue.cc:260] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d48129deb7b4c02b81063b6d9ebc562" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38037 } } peers { permanent_uuid: "1616528b7a99454abb6205c58f44d4e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46393 } } peers { permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 } }
I20260927 03:30:35.704553 15312 ts_tablet_manager.cc:1428] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260927 03:30:35.766451 15422 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:30:35.695098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.1:57037)
0927 03:30:35.695273 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:35.695316 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:35.695833 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:35.766204 (+ 70371us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
W20260927 03:30:35.766778 15420 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42247 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42247: Client connection negotiation failed: client connection to 127.0.98.62:42247: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 03:30:35.767171 15422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:35.767031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.1:35523)
0927 03:30:35.767087 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:35.767139 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 03:30:35.767534 15422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:35.767413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.1:33823)
0927 03:30:35.767483 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:35.767514 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 03:30:35.767660 15420 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 03:30:35.784179 15424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:30:35.784394 15424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:30:35.784417 15424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:30:35.785617 15424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:30:35.785660 15424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:30:35.785717 15424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:30:35.787163 15424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37079
--tserver_master_addrs=127.0.98.62:42247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:30:35.787402 15424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:30:35.787684 15424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:30:35.792593 15430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:35.792593 15433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:35.792610 15431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:35.792888 15424 server_base.cc:1034] running on GCE node
I20260927 03:30:35.793185 15424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:30:35.793391 15424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:30:35.794571 15424 hybrid_clock.cc:648] HybridClock initialized: now 1790479835794557 us; error 29 us; skew 500 ppm
Sep 27 03:30:35 dist-test-slave-j0hf krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:30:35.797123 15424 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:30:35.797610 15424 webserver.cc:460] Webserver started at http://127.0.98.2:37079/ using document root <none> and password file <none>
I20260927 03:30:35.797838 15424 fs_manager.cc:359] Metadata directory not provided
I20260927 03:30:35.797905 15424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:30:35.799252 15424 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:30:35.799929 15440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:35.800092 15424 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:35.800189 15424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "63bd9fc884d64ba6a55e09100051defc"
format_stamp: "Formatted at 2026-09-27 03:30:34 on dist-test-slave-j0hf"
server_key: "07f780196cf5073bfea103fbd09d8ee6"
server_key_iv: "0a1b79957bcffafaac6859c4313eb226"
server_key_version: "encryptionkey@0"
I20260927 03:30:35.800576 15424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:30:35.824004 15424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:30:35.825906 15424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:30:35.826079 15424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:30:35.826332 15424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:30:35.826653 15424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:30:35.826709 15424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:35.826753 15424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:30:35.826783 15424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:35.832748 15424 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39885
I20260927 03:30:35.832823 15553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39885 every 8 connection(s)
I20260927 03:30:35.833158 15424 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:30:35.834283 15556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:35.833839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.2:41873)
0927 03:30:35.834029 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:35.834137 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":76,"threads_started":1}
W20260927 03:30:35.834721 15554 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42247 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42247: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
I20260927 03:30:35.834986 15556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:35.834903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.2:37077)
0927 03:30:35.834943 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:35.834968 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 03:30:35.835222 15556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:35.835169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.2:33971)
0927 03:30:35.835190 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:35.835210 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260927 03:30:35.835330 15554 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 03:30:35.836643 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 15424
I20260927 03:30:35.836879 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38037
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42247
--webserver_port=38203
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:30:35.921586 15557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:30:35.921808 15557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:30:35.921832 15557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:30:35.922963 15557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:30:35.923000 15557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:30:35.923026 15557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:30:35.924517 15557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38203
--tserver_master_addrs=127.0.98.62:42247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:30:35.924726 15557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:30:35.924909 15557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:30:35.930099 15563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:35.930126 15562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:35.930289 15557 server_base.cc:1034] running on GCE node
W20260927 03:30:35.930264 15565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:35.930578 15557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:30:35.930822 15557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:30:35.931993 15557 hybrid_clock.cc:648] HybridClock initialized: now 1790479835931977 us; error 51 us; skew 500 ppm
Sep 27 03:30:35 dist-test-slave-j0hf krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:30:35.934499 15557 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:30:35.934930 15557 webserver.cc:460] Webserver started at http://127.0.98.3:38203/ using document root <none> and password file <none>
I20260927 03:30:35.935184 15557 fs_manager.cc:359] Metadata directory not provided
I20260927 03:30:35.935252 15557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:30:35.936625 15557 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:30:35.937199 15572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:35.937361 15557 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:30:35.937455 15557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "1d48129deb7b4c02b81063b6d9ebc562"
format_stamp: "Formatted at 2026-09-27 03:30:34 on dist-test-slave-j0hf"
server_key: "d466fd0bfc3ea9b165c5dfcad1a18e93"
server_key_iv: "ac2decac81bd1c7a4ed079e0ca8ffb83"
server_key_version: "encryptionkey@0"
I20260927 03:30:35.937832 15557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:30:35.982901 15557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:30:35.984910 15557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:30:35.985083 15557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:30:35.985327 15557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:30:35.985637 15557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:30:35.985692 15557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:35.985733 15557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:30:35.985760 15557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:35.991364 15557 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38037
I20260927 03:30:35.991470 15685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38037 every 8 connection(s)
I20260927 03:30:35.991797 15557 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:30:35.992231 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 15557
I20260927 03:30:35.992982 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:35.992596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.3:42463)
0927 03:30:35.992765 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:35.992846 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
W20260927 03:30:35.993337 15686 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42247 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42247: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
I20260927 03:30:35.993602 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:35.993508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.3:52051)
0927 03:30:35.993558 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:35.993585 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 03:30:35.993906 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:35.993819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.3:37157)
0927 03:30:35.993860 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:35.993882 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 03:30:35.994004 15686 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 03:30:36.768289 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:36.768058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.1:39343)
0927 03:30:36.768185 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:36.768225 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
I20260927 03:30:36.835927 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:36.835715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.2:51125)
0927 03:30:36.835844 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:36.835881 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20260927 03:30:36.994582 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:36.994340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.3:42491)
0927 03:30:36.994485 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:36.994523 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":57,"threads_started":1}
I20260927 03:30:37.284042 15693 raft_consensus.cc:482] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:30:37.284202 15693 raft_consensus.cc:504] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d48129deb7b4c02b81063b6d9ebc562" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38037 } } peers { permanent_uuid: "1616528b7a99454abb6205c58f44d4e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46393 } } peers { permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 } }
I20260927 03:30:37.284541 15693 leader_election.cc:288] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d48129deb7b4c02b81063b6d9ebc562 (127.0.98.3:38037), 63bd9fc884d64ba6a55e09100051defc (127.0.98.2:39885)
Sep 27 03:30:37 dist-test-slave-j0hf krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 27 03:30:37 dist-test-slave-j0hf krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 03:30:37.304235 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:37.284907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36627 (local address 127.0.98.3:38037)
0927 03:30:37.285071 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:37.285084 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:37.285132 (+ 48us) server_negotiation.cc:407] Connection header received
0927 03:30:37.285817 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:37.285819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:37.285862 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:37.286000 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:37.286769 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:37.287706 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:37.288713 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:37.289085 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:37.289227 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:37.300285 (+ 11058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:37.300290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:37.300290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:37.300300 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:37.302257 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:37.302869 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:37.302872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:37.302873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:37.302959 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:37.303255 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:37.303255 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:37.303256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:37.303643 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:37.303746 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:37.303980 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 03:30:37.304108 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":66,"threads_started":1}
I20260927 03:30:37.304394 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:37.284779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38037 (local address 127.0.98.1:36627)
0927 03:30:37.285069 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:37.285082 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:37.285146 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:37.286054 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:37.286057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:37.286441 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:30:37.286637 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:37.286642 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:37.287799 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:37.287801 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:37.288588 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:37.288591 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:37.289189 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:37.289190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:37.289282 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:37.298497 (+ 9215us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:30:37.298504 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:30:37.300130 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:30:37.302347 (+ 2217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:37.302354 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:37.302356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:37.302781 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:37.303033 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:37.303034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:37.303034 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:37.303151 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:37.303731 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:30:37.303732 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:30:37.303835 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 03:30:37.303927 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 03:30:37.304118 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":56,"thread_start_us":93,"threads_started":1}
I20260927 03:30:37.304564 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:37.285010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39885 (local address 127.0.98.1:52079)
0927 03:30:37.285151 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:37.285162 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:37.285229 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:37.286056 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:37.286058 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:37.286440 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:30:37.286637 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:37.286642 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:37.287828 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:37.287829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:37.288586 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:37.288588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:37.289164 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:37.289165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:37.289282 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:37.298492 (+ 9210us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:30:37.298502 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:30:37.300566 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:30:37.302492 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:37.302493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:37.302494 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:37.302781 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:37.303065 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:37.303066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:37.303066 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:37.303195 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:37.303690 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:30:37.303697 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:30:37.303823 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 03:30:37.303928 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 03:30:37.304222 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":145,"thread_start_us":40,"threads_started":1}
I20260927 03:30:37.304273 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:37.285245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52079 (local address 127.0.98.2:39885)
0927 03:30:37.285380 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:37.285389 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:37.285420 (+ 31us) server_negotiation.cc:407] Connection header received
0927 03:30:37.285845 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:37.285847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:37.285883 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:37.286032 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:37.286794 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:37.287711 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:37.288671 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:37.289074 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:37.289190 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:37.300691 (+ 11501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:37.300695 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:37.300695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:37.300705 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:37.302378 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:37.302889 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:37.302892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:37.302893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:37.302985 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:37.303255 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:37.303256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:37.303256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:37.303609 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:37.303710 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:37.303981 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 03:30:37.304115 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
W20260927 03:30:37.305356 15307 leader_election.cc:341] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d48129deb7b4c02b81063b6d9ebc562 (127.0.98.3:38037): Not found: Tablet not found: 2715d6ed3ba64bc4bc1b0875ccf0aecd
W20260927 03:30:37.305545 15307 leader_election.cc:341] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63bd9fc884d64ba6a55e09100051defc (127.0.98.2:39885): Not found: Tablet not found: 2715d6ed3ba64bc4bc1b0875ccf0aecd
I20260927 03:30:37.305605 15307 leader_election.cc:302] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1616528b7a99454abb6205c58f44d4e9; no voters: 1d48129deb7b4c02b81063b6d9ebc562, 63bd9fc884d64ba6a55e09100051defc
I20260927 03:30:37.305694 15693 raft_consensus.cc:2726] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:30:37.768898 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:37.768762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.1:44277)
0927 03:30:37.768823 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:37.768857 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 03:30:37.836741 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:37.836507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.2:35167)
0927 03:30:37.836658 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:37.836699 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":70,"threads_started":1}
I20260927 03:30:37.995283 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:37.995093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.3:55885)
0927 03:30:37.995214 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:37.995249 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260927 03:30:38.769591 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:38.769377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.1:49505)
0927 03:30:38.769504 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:38.769539 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
I20260927 03:30:38.837523 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:38.837291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.2:39659)
0927 03:30:38.837424 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:38.837476 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
I20260927 03:30:38.996047 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:38.995821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.3:34631)
0927 03:30:38.995958 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:38.995998 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260927 03:30:39.268424 15703 raft_consensus.cc:482] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:30:39.268571 15703 raft_consensus.cc:504] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d48129deb7b4c02b81063b6d9ebc562" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38037 } } peers { permanent_uuid: "1616528b7a99454abb6205c58f44d4e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46393 } } peers { permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 } }
I20260927 03:30:39.268783 15703 leader_election.cc:288] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d48129deb7b4c02b81063b6d9ebc562 (127.0.98.3:38037), 63bd9fc884d64ba6a55e09100051defc (127.0.98.2:39885)
W20260927 03:30:39.269240 15307 leader_election.cc:341] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d48129deb7b4c02b81063b6d9ebc562 (127.0.98.3:38037): Not found: Tablet not found: 2715d6ed3ba64bc4bc1b0875ccf0aecd
W20260927 03:30:39.269304 15307 leader_election.cc:341] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63bd9fc884d64ba6a55e09100051defc (127.0.98.2:39885): Not found: Tablet not found: 2715d6ed3ba64bc4bc1b0875ccf0aecd
I20260927 03:30:39.269322 15307 leader_election.cc:302] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1616528b7a99454abb6205c58f44d4e9; no voters: 1d48129deb7b4c02b81063b6d9ebc562, 63bd9fc884d64ba6a55e09100051defc
I20260927 03:30:39.269387 15703 raft_consensus.cc:2726] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:30:39.770354 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:39.770143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.1:46569)
0927 03:30:39.770263 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:39.770306 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260927 03:30:39.838212 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:39.837990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.2:56929)
0927 03:30:39.838131 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:39.838166 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":57,"threads_started":1}
I20260927 03:30:39.996773 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:39.996531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.3:49741)
0927 03:30:39.996690 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:39.996729 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":79,"threads_started":1}
I20260927 03:30:40.771162 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:40.770919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.1:38977)
0927 03:30:40.771070 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:40.771111 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":86,"threads_started":1}
I20260927 03:30:40.838963 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:40.838754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.2:59623)
0927 03:30:40.838879 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:40.838917 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260927 03:30:40.997488 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:40.997265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.3:57645)
0927 03:30:40.997405 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:40.997443 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":69,"threads_started":1}
I20260927 03:30:41.578053 15710 raft_consensus.cc:482] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:30:41.578173 15710 raft_consensus.cc:504] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d48129deb7b4c02b81063b6d9ebc562" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38037 } } peers { permanent_uuid: "1616528b7a99454abb6205c58f44d4e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46393 } } peers { permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 } }
I20260927 03:30:41.578377 15710 leader_election.cc:288] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d48129deb7b4c02b81063b6d9ebc562 (127.0.98.3:38037), 63bd9fc884d64ba6a55e09100051defc (127.0.98.2:39885)
W20260927 03:30:41.578859 15307 leader_election.cc:341] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63bd9fc884d64ba6a55e09100051defc (127.0.98.2:39885): Not found: Tablet not found: 2715d6ed3ba64bc4bc1b0875ccf0aecd
W20260927 03:30:41.578930 15307 leader_election.cc:341] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d48129deb7b4c02b81063b6d9ebc562 (127.0.98.3:38037): Not found: Tablet not found: 2715d6ed3ba64bc4bc1b0875ccf0aecd
I20260927 03:30:41.579035 15307 leader_election.cc:302] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1616528b7a99454abb6205c58f44d4e9; no voters: 1d48129deb7b4c02b81063b6d9ebc562, 63bd9fc884d64ba6a55e09100051defc
I20260927 03:30:41.579175 15710 raft_consensus.cc:2726] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:30:41.771948 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:41.771695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.1:46137)
0927 03:30:41.771846 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:41.771891 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":64,"threads_started":1}
I20260927 03:30:41.839717 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:41.839510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.2:47521)
0927 03:30:41.839634 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:41.839672 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":59,"threads_started":1}
I20260927 03:30:41.998234 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:41.998026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.3:34785)
0927 03:30:41.998159 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:41.998194 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":56,"threads_started":1}
I20260927 03:30:42.772677 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:42.772460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.1:47597)
0927 03:30:42.772589 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:42.772628 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":67,"threads_started":1}
I20260927 03:30:42.840468 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:42.840264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.2:45795)
0927 03:30:42.840385 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:42.840418 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":54,"threads_started":1}
I20260927 03:30:42.998875 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:42.998658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.3:45899)
0927 03:30:42.998799 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:42.998835 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":69,"threads_started":1}
I20260927 03:30:43.398808 15717 raft_consensus.cc:482] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:30:43.398901 15717 raft_consensus.cc:504] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d48129deb7b4c02b81063b6d9ebc562" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38037 } } peers { permanent_uuid: "1616528b7a99454abb6205c58f44d4e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46393 } } peers { permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 } }
I20260927 03:30:43.399022 15717 leader_election.cc:288] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d48129deb7b4c02b81063b6d9ebc562 (127.0.98.3:38037), 63bd9fc884d64ba6a55e09100051defc (127.0.98.2:39885)
W20260927 03:30:43.399540 15307 leader_election.cc:341] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d48129deb7b4c02b81063b6d9ebc562 (127.0.98.3:38037): Not found: Tablet not found: 2715d6ed3ba64bc4bc1b0875ccf0aecd
W20260927 03:30:43.399609 15307 leader_election.cc:341] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63bd9fc884d64ba6a55e09100051defc (127.0.98.2:39885): Not found: Tablet not found: 2715d6ed3ba64bc4bc1b0875ccf0aecd
I20260927 03:30:43.399626 15307 leader_election.cc:302] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1616528b7a99454abb6205c58f44d4e9; no voters: 1d48129deb7b4c02b81063b6d9ebc562, 63bd9fc884d64ba6a55e09100051defc
I20260927 03:30:43.399745 15717 raft_consensus.cc:2726] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:30:43.773612 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:43.773347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.1:53929)
0927 03:30:43.773519 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:43.773552 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":91,"threads_started":1}
I20260927 03:30:43.841166 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:43.840913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.2:51489)
0927 03:30:43.841031 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:43.841099 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
I20260927 03:30:43.999668 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:43.999395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.3:60801)
0927 03:30:43.999579 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:43.999621 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":113,"threads_started":1}
I20260927 03:30:44.774420 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:44.774186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.1:57123)
0927 03:30:44.774326 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:44.774371 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":61,"threads_started":1}
I20260927 03:30:44.841892 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:44.841624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.2:43819)
0927 03:30:44.841781 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:44.841834 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":82,"threads_started":1}
I20260927 03:30:45.000442 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:45.000236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.3:38009)
0927 03:30:45.000360 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:45.000400 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260927 03:30:45.775257 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:45.775021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.1:55095)
0927 03:30:45.775161 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:45.775207 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":68,"threads_started":1}
I20260927 03:30:45.842684 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:45.842366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.2:56405)
0927 03:30:45.842579 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:45.842619 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":59,"threads_started":1}
I20260927 03:30:46.001173 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:46.000923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.3:46851)
0927 03:30:46.001055 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:46.001093 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
I20260927 03:30:46.776080 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:46.775824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.1:41387)
0927 03:30:46.775976 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:46.776022 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
I20260927 03:30:46.843346 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:46.843133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.2:56391)
0927 03:30:46.843255 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:46.843297 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260927 03:30:47.001879 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:47.001664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.3:57543)
0927 03:30:47.001801 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:47.001834 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":70,"threads_started":1}
I20260927 03:30:47.776849 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:47.776602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.1:52197)
0927 03:30:47.776755 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:47.776799 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":78,"threads_started":1}
I20260927 03:30:47.844074 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:47.843842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.2:52201)
0927 03:30:47.843979 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:47.844025 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":62,"threads_started":1}
I20260927 03:30:48.002743 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:48.002484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.3:34547)
0927 03:30:48.002651 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:48.002694 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":81,"threads_started":1}
I20260927 03:30:48.777614 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:48.777404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.1:39017)
0927 03:30:48.777531 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:48.777565 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260927 03:30:48.844921 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:48.844669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.2:54763)
0927 03:30:48.844829 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:48.844872 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":81,"threads_started":1}
I20260927 03:30:49.003607 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:49.003272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.3:47497)
0927 03:30:49.003500 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:49.003555 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":76,"threads_started":1}
I20260927 03:30:49.778347 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:49.778127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.1:47755)
0927 03:30:49.778254 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:49.778292 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20260927 03:30:49.845710 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:49.845482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.2:51705)
0927 03:30:49.845621 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:49.845663 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260927 03:30:50.004429 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:50.004109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.3:42307)
0927 03:30:50.004304 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:50.004333 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":150,"threads_started":1}
I20260927 03:30:50.479476 15739 raft_consensus.cc:482] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:30:50.479579 15739 raft_consensus.cc:504] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d48129deb7b4c02b81063b6d9ebc562" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38037 } } peers { permanent_uuid: "1616528b7a99454abb6205c58f44d4e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46393 } } peers { permanent_uuid: "63bd9fc884d64ba6a55e09100051defc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39885 } }
I20260927 03:30:50.479753 15739 leader_election.cc:288] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d48129deb7b4c02b81063b6d9ebc562 (127.0.98.3:38037), 63bd9fc884d64ba6a55e09100051defc (127.0.98.2:39885)
W20260927 03:30:50.480211 15307 leader_election.cc:341] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63bd9fc884d64ba6a55e09100051defc (127.0.98.2:39885): Not found: Tablet not found: 2715d6ed3ba64bc4bc1b0875ccf0aecd
W20260927 03:30:50.480304 15307 leader_election.cc:341] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d48129deb7b4c02b81063b6d9ebc562 (127.0.98.3:38037): Not found: Tablet not found: 2715d6ed3ba64bc4bc1b0875ccf0aecd
I20260927 03:30:50.480324 15307 leader_election.cc:302] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1616528b7a99454abb6205c58f44d4e9; no voters: 1d48129deb7b4c02b81063b6d9ebc562, 63bd9fc884d64ba6a55e09100051defc
I20260927 03:30:50.480450 15739 raft_consensus.cc:2726] T 2715d6ed3ba64bc4bc1b0875ccf0aecd P 1616528b7a99454abb6205c58f44d4e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:30:50.779163 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:50.778935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.1:34901)
0927 03:30:50.779078 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:50.779111 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":61,"threads_started":1}
I20260927 03:30:50.846544 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:50.846319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.2:57369)
0927 03:30:50.846462 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:50.846499 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
I20260927 03:30:50.997711 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 15261
I20260927 03:30:51.002935 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 15424
I20260927 03:30:51.005201 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:51.004957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42247 (local address 127.0.98.3:49143)
0927 03:30:51.005093 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:51.005139 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":56,"threads_started":1}
I20260927 03:30:51.007684 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 15557
2026-09-27T03:30:51Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18327 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:30:51 dist-test-slave-j0hf krb5kdc[15745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:30:51 dist-test-slave-j0hf krb5kdc[15745](info): set up 2 sockets
Sep 27 03:30:51 dist-test-slave-j0hf krb5kdc[15745](info): commencing operation
krb5kdc: starting...
W20260927 03:30:52.067698 392 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:30:52 dist-test-slave-j0hf krb5kdc[15745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:30:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:30:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:30:52.186081 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35465
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:30:52.265681 15758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:30:52.265923 15758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:30:52.265956 15758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:30:52.267099 15758 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 03:30:52.267167 15758 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 03:30:52.267189 15758 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 03:30:52.267206 15758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:30:52.267228 15758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:30:52.267259 15758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:30:52.267283 15758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:30:52.267313 15758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:30:52.268821 15758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:30:52.269099 15758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:30:52.269327 15758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:30:52.274624 15766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:52.274649 15763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:52.274724 15764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:52.274791 15758 server_base.cc:1034] running on GCE node
I20260927 03:30:52.275161 15758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:30:52.275588 15758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:30:52.276773 15758 hybrid_clock.cc:648] HybridClock initialized: now 1790479852276757 us; error 28 us; skew 500 ppm
Sep 27 03:30:52 dist-test-slave-j0hf krb5kdc[15745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:30:52.279477 15758 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:30:52.280002 15758 webserver.cc:460] Webserver started at http://127.0.98.62:36779/ using document root <none> and password file <none>
I20260927 03:30:52.280263 15758 fs_manager.cc:359] Metadata directory not provided
I20260927 03:30:52.280333 15758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:30:52.280500 15758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:30:52.281646 15758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "ac032e00c98e4600a63fd0da0e315327"
format_stamp: "Formatted at 2026-09-27 03:30:52 on dist-test-slave-j0hf"
server_key: "5ee7528513c5b10ecf32bf3c3eea2548"
server_key_iv: "8abe229cc80596e3e7a28fdbc1e9bb3c"
server_key_version: "encryptionkey@0"
I20260927 03:30:52.282025 15758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "ac032e00c98e4600a63fd0da0e315327"
format_stamp: "Formatted at 2026-09-27 03:30:52 on dist-test-slave-j0hf"
server_key: "5ee7528513c5b10ecf32bf3c3eea2548"
server_key_iv: "8abe229cc80596e3e7a28fdbc1e9bb3c"
server_key_version: "encryptionkey@0"
I20260927 03:30:52.283375 15758 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:30:52.284139 15773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:52.284320 15758 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:30:52.284418 15758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "ac032e00c98e4600a63fd0da0e315327"
format_stamp: "Formatted at 2026-09-27 03:30:52 on dist-test-slave-j0hf"
server_key: "5ee7528513c5b10ecf32bf3c3eea2548"
server_key_iv: "8abe229cc80596e3e7a28fdbc1e9bb3c"
server_key_version: "encryptionkey@0"
I20260927 03:30:52.284507 15758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:30:52.294018 15758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:30:52.295956 15758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:30:52.296125 15758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:30:52.300127 15758 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35465
I20260927 03:30:52.300173 15825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35465 every 8 connection(s)
I20260927 03:30:52.300530 15758 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:30:52.300669 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 15758
I20260927 03:30:52.300777 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:30:52.301011 392 external_mini_cluster.cc:1428] Setting key 74cd78af39ef9b24e518951614c00f62
I20260927 03:30:52.301560 15826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:30:52.304230 15826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327: Bootstrap starting.
I20260927 03:30:52.304848 15826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327: Neither blocks nor log segments found. Creating new log.
I20260927 03:30:52.305137 15826 log.cc:824] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327: Log is configured to *not* fsync() on all Append() calls
I20260927 03:30:52.305795 15826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327: No bootstrap required, opened a new log
Sep 27 03:30:52 dist-test-slave-j0hf krb5kdc[15745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:30:52.307703 15826 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac032e00c98e4600a63fd0da0e315327" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35465 } }
I20260927 03:30:52.307878 15826 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:30:52.307927 15826 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac032e00c98e4600a63fd0da0e315327, State: Initialized, Role: FOLLOWER
I20260927 03:30:52.308068 15826 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac032e00c98e4600a63fd0da0e315327" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35465 } }
I20260927 03:30:52.308162 15826 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:30:52.308205 15826 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:30:52.308252 15826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:30:52.308820 15826 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac032e00c98e4600a63fd0da0e315327" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35465 } }
I20260927 03:30:52.308933 15826 leader_election.cc:302] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac032e00c98e4600a63fd0da0e315327; no voters:
I20260927 03:30:52.309095 15826 leader_election.cc:288] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:30:52.309121 15831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:30:52.309332 15831 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [term 1 LEADER]: Becoming Leader. State: Replica: ac032e00c98e4600a63fd0da0e315327, State: Running, Role: LEADER
I20260927 03:30:52.309363 15826 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:30:52.309425 15831 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac032e00c98e4600a63fd0da0e315327" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35465 } }
I20260927 03:30:52.309770 15832 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac032e00c98e4600a63fd0da0e315327" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac032e00c98e4600a63fd0da0e315327" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35465 } } }
I20260927 03:30:52.309923 15832 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [sys.catalog]: This master's current role is: LEADER
I20260927 03:30:52.310112 15832 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac032e00c98e4600a63fd0da0e315327. Latest consensus state: current_term: 1 leader_uuid: "ac032e00c98e4600a63fd0da0e315327" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac032e00c98e4600a63fd0da0e315327" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35465 } } }
I20260927 03:30:52.310178 15832 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [sys.catalog]: This master's current role is: LEADER
W20260927 03:30:52.311571 15848 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:30:52.311666 15848 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:30:52.311745 15836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:30:52.311892 15836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:30:52.312047 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:52.301561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44820 (local address 127.0.98.62:35465)
0927 03:30:52.301747 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:52.301756 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:52.301789 (+ 33us) server_negotiation.cc:407] Connection header received
0927 03:30:52.302221 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:52.302228 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:52.302271 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:52.302419 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:52.303064 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:52.304039 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:52.304892 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:52.305370 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:52.305486 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:52.307889 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:52.307892 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:52.307893 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:52.307922 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:52.310070 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:52.310539 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:52.310542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:52.310543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:52.310657 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:52.310905 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:52.310906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:52.310906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:52.311240 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:52.311419 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:52.311557 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 03:30:52.311787 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":85,"threads_started":1}
I20260927 03:30:52.313388 15836 catalog_manager.cc:1347] Generated new cluster ID: d2c2c459365d4fe2aeda94b99a865a1f
I20260927 03:30:52.313439 15836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:30:52.328823 15836 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:30:52.329447 15836 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:30:52.335393 15836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327: Generated new TSK 0
I20260927 03:30:52.335629 15836 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:30:52.362576 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35465
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:30:52.444972 15853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:30:52.445194 15853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:30:52.445218 15853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:30:52.446412 15853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:30:52.446457 15853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:30:52.446484 15853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:30:52.448110 15853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:30:52.448376 15853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:30:52.448618 15853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:30:52.453682 15859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:52.453650 15858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:52.453665 15861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:52.453991 15853 server_base.cc:1034] running on GCE node
I20260927 03:30:52.454200 15853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:30:52.454447 15853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:30:52.455605 15853 hybrid_clock.cc:648] HybridClock initialized: now 1790479852455601 us; error 37 us; skew 500 ppm
Sep 27 03:30:52 dist-test-slave-j0hf krb5kdc[15745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:30:52.458163 15853 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:30:52.458549 15853 webserver.cc:460] Webserver started at http://127.0.98.1:44781/ using document root <none> and password file <none>
I20260927 03:30:52.458724 15853 fs_manager.cc:359] Metadata directory not provided
I20260927 03:30:52.458763 15853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:30:52.458851 15853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:30:52.459923 15853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/data/instance:
uuid: "b77d4b143c8d4406881fc7c1654dbf61"
format_stamp: "Formatted at 2026-09-27 03:30:52 on dist-test-slave-j0hf"
server_key: "c90e9ad6f2a4c8b27674a814971a62df"
server_key_iv: "acc6c050ad6c1c9f35c598926bb92998"
server_key_version: "encryptionkey@0"
I20260927 03:30:52.460217 15853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b77d4b143c8d4406881fc7c1654dbf61"
format_stamp: "Formatted at 2026-09-27 03:30:52 on dist-test-slave-j0hf"
server_key: "c90e9ad6f2a4c8b27674a814971a62df"
server_key_iv: "acc6c050ad6c1c9f35c598926bb92998"
server_key_version: "encryptionkey@0"
I20260927 03:30:52.461448 15853 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:30:52.462357 15868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:52.462535 15853 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:30:52.462618 15853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "b77d4b143c8d4406881fc7c1654dbf61"
format_stamp: "Formatted at 2026-09-27 03:30:52 on dist-test-slave-j0hf"
server_key: "c90e9ad6f2a4c8b27674a814971a62df"
server_key_iv: "acc6c050ad6c1c9f35c598926bb92998"
server_key_version: "encryptionkey@0"
I20260927 03:30:52.462692 15853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:30:52.484730 15853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:30:52.487285 15853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:30:52.487488 15853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:30:52.487751 15853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:30:52.488085 15853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:30:52.488139 15853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:52.488180 15853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:30:52.488210 15853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:52.494017 15853 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38477
I20260927 03:30:52.494066 15981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38477 every 8 connection(s)
I20260927 03:30:52.494400 15853 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:30:52.497954 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 15853
I20260927 03:30:52.498081 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:30:52.498286 392 external_mini_cluster.cc:1428] Setting key e324b0fcd88ee2985c5e823ebd3048f5
Sep 27 03:30:52 dist-test-slave-j0hf krb5kdc[15745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:30:52.505115 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:52.495125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60543 (local address 127.0.98.62:35465)
0927 03:30:52.495189 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:52.495191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:52.495393 (+ 202us) server_negotiation.cc:407] Connection header received
0927 03:30:52.495929 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:52.495930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:52.495939 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:52.495995 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:52.496930 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:52.497453 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:52.498840 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:52.499128 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:52.499183 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:52.502154 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:52.502155 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:52.502155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:52.502162 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:52.503641 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:52.504144 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:52.504145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:52.504145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:52.504194 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:52.504492 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:52.504492 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:52.504493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:52.504593 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:52.504642 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:52.504982 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 03:30:52.505049 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:30:52.505393 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:52.495092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.1:60543)
0927 03:30:52.495253 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:52.495296 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:52.495827 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:52.496058 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:52.496061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:52.496632 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:30:52.496842 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:52.496847 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:52.497531 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:52.497531 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:52.498761 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:52.498764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:52.499216 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:52.499217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:52.499380 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:52.499927 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:30:52.499937 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:30:52.502063 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:30:52.503717 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:52.503724 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:52.503727 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:52.504065 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:52.504266 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:52.504267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:52.504267 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:52.504422 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:52.504660 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:30:52.504666 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:30:52.504835 (+ 169us) client_negotiation.cc:769] Sending connection context
0927 03:30:52.504966 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 03:30:52.505141 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260927 03:30:52.505973 15982 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35465
I20260927 03:30:52.506072 15982 heartbeater.cc:464] Registering TS with master...
I20260927 03:30:52.506301 15982 heartbeater.cc:511] Master 127.0.98.62:35465 requested a full tablet report, sending...
I20260927 03:30:52.506827 15790 ts_manager.cc:194] Registered new tserver with Master: b77d4b143c8d4406881fc7c1654dbf61 (127.0.98.1:38477)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 03:30:52.507408 15790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60543
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:30:52.524292 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35465
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:30:52.606357 15987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:30:52.606630 15987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:30:52.606673 15987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:30:52.607905 15987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:30:52.607986 15987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:30:52.608033 15987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:30:52.609483 15987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:30:52.609812 15987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:30:52.610093 15987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:30:52.615051 15992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:52.615061 15993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:52.615061 15995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:52.615298 15987 server_base.cc:1034] running on GCE node
I20260927 03:30:52.615522 15987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:30:52.615720 15987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:30:52.616861 15987 hybrid_clock.cc:648] HybridClock initialized: now 1790479852616850 us; error 26 us; skew 500 ppm
Sep 27 03:30:52 dist-test-slave-j0hf krb5kdc[15745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:30:52.619477 15987 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:30:52.619930 15987 webserver.cc:460] Webserver started at http://127.0.98.2:33211/ using document root <none> and password file <none>
I20260927 03:30:52.620165 15987 fs_manager.cc:359] Metadata directory not provided
I20260927 03:30:52.620235 15987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:30:52.620389 15987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:30:52.621454 15987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/data/instance:
uuid: "33fa80feb50e4c3eb17927e5ccc7142f"
format_stamp: "Formatted at 2026-09-27 03:30:52 on dist-test-slave-j0hf"
server_key: "1a539d321fc934b389ee8dd631a6767e"
server_key_iv: "f88e40753c245127a2457df7b6f29ef6"
server_key_version: "encryptionkey@0"
I20260927 03:30:52.621830 15987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/wal/instance:
uuid: "33fa80feb50e4c3eb17927e5ccc7142f"
format_stamp: "Formatted at 2026-09-27 03:30:52 on dist-test-slave-j0hf"
server_key: "1a539d321fc934b389ee8dd631a6767e"
server_key_iv: "f88e40753c245127a2457df7b6f29ef6"
server_key_version: "encryptionkey@0"
I20260927 03:30:52.623132 15987 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:30:52.623941 16002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:52.624107 15987 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:52.624213 15987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "33fa80feb50e4c3eb17927e5ccc7142f"
format_stamp: "Formatted at 2026-09-27 03:30:52 on dist-test-slave-j0hf"
server_key: "1a539d321fc934b389ee8dd631a6767e"
server_key_iv: "f88e40753c245127a2457df7b6f29ef6"
server_key_version: "encryptionkey@0"
I20260927 03:30:52.624292 15987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:30:52.652201 15987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:30:52.654307 15987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:30:52.654485 15987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:30:52.654732 15987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:30:52.655045 15987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:30:52.655102 15987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:52.655143 15987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:30:52.655170 15987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:52.661037 15987 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38421
I20260927 03:30:52.661080 16115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38421 every 8 connection(s)
I20260927 03:30:52.661446 15987 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:30:52 dist-test-slave-j0hf krb5kdc[15745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:30:52.669697 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 15987
I20260927 03:30:52.669855 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:30:52.670094 392 external_mini_cluster.cc:1428] Setting key 3079b71835e31e99a3c4a7fc1b8c5c54
I20260927 03:30:52.671154 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:52.662272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41911 (local address 127.0.98.62:35465)
0927 03:30:52.662361 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:52.662363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:52.662519 (+ 156us) server_negotiation.cc:407] Connection header received
0927 03:30:52.663045 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:52.663046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:52.663058 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:52.663114 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:52.664088 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:52.664590 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:52.665677 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:52.665883 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:52.665968 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:52.668215 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:52.668216 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:52.668217 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:52.668225 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:52.669639 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:52.670140 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:52.670141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:52.670142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:52.670205 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:52.670466 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:52.670467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:52.670467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:52.670585 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:52.670662 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:52.670981 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 03:30:52.671060 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 03:30:52.671371 16118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:52.662203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.2:41911)
0927 03:30:52.662384 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:52.662432 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:52.662938 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:52.663196 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:52.663199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:52.663795 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:30:52.663991 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:52.663995 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:52.664711 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:52.664712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:52.665591 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:52.665593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:52.665976 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:52.665977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:52.666104 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:52.666674 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:30:52.666682 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:30:52.668116 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:30:52.669742 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:52.669750 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:52.669752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:52.670052 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:52.670258 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:52.670259 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:52.670259 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:52.670408 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:52.670653 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:30:52.670660 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:30:52.670829 (+ 169us) client_negotiation.cc:769] Sending connection context
0927 03:30:52.670961 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 03:30:52.671124 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
I20260927 03:30:52.671877 16116 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35465
I20260927 03:30:52.671959 16116 heartbeater.cc:464] Registering TS with master...
I20260927 03:30:52.672155 16116 heartbeater.cc:511] Master 127.0.98.62:35465 requested a full tablet report, sending...
I20260927 03:30:52.672595 15790 ts_manager.cc:194] Registered new tserver with Master: 33fa80feb50e4c3eb17927e5ccc7142f (127.0.98.2:38421)
I20260927 03:30:52.673008 15790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41911
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:30:52.696489 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35465
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:30:52.778637 16121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:30:52.778908 16121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:30:52.778952 16121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:30:52.780243 16121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:30:52.780313 16121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:30:52.780359 16121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:30:52.781840 16121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:30:52.782112 16121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:30:52.782346 16121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:30:52.787389 16126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:30:52.787390 16129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:52.787581 16121 server_base.cc:1034] running on GCE node
W20260927 03:30:52.787724 16127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:30:52.787984 16121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:30:52.788245 16121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:30:52.789395 16121 hybrid_clock.cc:648] HybridClock initialized: now 1790479852789384 us; error 28 us; skew 500 ppm
Sep 27 03:30:52 dist-test-slave-j0hf krb5kdc[15745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:30:52.791965 16121 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:30:52.792399 16121 webserver.cc:460] Webserver started at http://127.0.98.3:42765/ using document root <none> and password file <none>
I20260927 03:30:52.792621 16121 fs_manager.cc:359] Metadata directory not provided
I20260927 03:30:52.792686 16121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:30:52.792836 16121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:30:52.793869 16121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/data/instance:
uuid: "cb8b2d9f64ae44a0a6c65087194bf770"
format_stamp: "Formatted at 2026-09-27 03:30:52 on dist-test-slave-j0hf"
server_key: "96524ede5ba89976c92496a281b35074"
server_key_iv: "3998a782d2aba15a960a0a4bd54ffede"
server_key_version: "encryptionkey@0"
I20260927 03:30:52.794234 16121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cb8b2d9f64ae44a0a6c65087194bf770"
format_stamp: "Formatted at 2026-09-27 03:30:52 on dist-test-slave-j0hf"
server_key: "96524ede5ba89976c92496a281b35074"
server_key_iv: "3998a782d2aba15a960a0a4bd54ffede"
server_key_version: "encryptionkey@0"
I20260927 03:30:52.795596 16121 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:30:52.796345 16136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:52.796552 16121 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:30:52.796649 16121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "cb8b2d9f64ae44a0a6c65087194bf770"
format_stamp: "Formatted at 2026-09-27 03:30:52 on dist-test-slave-j0hf"
server_key: "96524ede5ba89976c92496a281b35074"
server_key_iv: "3998a782d2aba15a960a0a4bd54ffede"
server_key_version: "encryptionkey@0"
I20260927 03:30:52.796736 16121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:30:52.804214 16121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:30:52.806445 16121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:30:52.806602 16121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:30:52.806831 16121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:30:52.807124 16121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:30:52.807176 16121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:52.807219 16121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:30:52.807246 16121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:30:52.813349 16121 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33805
I20260927 03:30:52.813496 16249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33805 every 8 connection(s)
I20260927 03:30:52.813760 16121 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:30:52 dist-test-slave-j0hf krb5kdc[15745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:30:52.821409 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 16121
I20260927 03:30:52.821516 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:30:52.821715 392 external_mini_cluster.cc:1428] Setting key bc7864f47182b35ce30ebc88ab997a5e
I20260927 03:30:52.823525 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:52.814493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47065 (local address 127.0.98.62:35465)
0927 03:30:52.814558 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:52.814560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:52.814790 (+ 230us) server_negotiation.cc:407] Connection header received
0927 03:30:52.815283 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:52.815283 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:52.815294 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:52.815355 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:52.816322 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:52.816809 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:52.817954 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:52.818141 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:52.818196 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:52.820477 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:52.820479 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:52.820479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:52.820485 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:52.822098 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:52.822574 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:52.822575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:52.822575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:52.822639 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:52.822891 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:52.822891 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:52.822892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:52.822996 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:52.823046 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:52.823324 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 03:30:52.823389 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 03:30:52.823721 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:52.814497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.3:47065)
0927 03:30:52.814663 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:52.814700 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:52.815162 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:52.815413 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:52.815416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:52.816039 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:30:52.816242 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:52.816247 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:52.816911 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:52.816912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:52.817873 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:52.817875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:52.818214 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:52.818215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:52.818394 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:52.818881 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:30:52.818889 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:30:52.820381 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:30:52.822175 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:52.822182 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:52.822184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:52.822498 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:52.822713 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:30:52.822713 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:30:52.822714 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:30:52.822828 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:30:52.823054 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:30:52.823059 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:30:52.823212 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 03:30:52.823300 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 03:30:52.823479 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":91,"threads_started":1}
I20260927 03:30:52.824156 16250 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35465
I20260927 03:30:52.824268 16250 heartbeater.cc:464] Registering TS with master...
I20260927 03:30:52.824446 16250 heartbeater.cc:511] Master 127.0.98.62:35465 requested a full tablet report, sending...
I20260927 03:30:52.824805 15790 ts_manager.cc:194] Registered new tserver with Master: cb8b2d9f64ae44a0a6c65087194bf770 (127.0.98.3:33805)
I20260927 03:30:52.825181 15790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47065
I20260927 03:30:52.833563 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:30:52.841020 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:52.834918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44826 (local address 127.0.98.62:35465)
0927 03:30:52.835008 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:52.835010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:52.835105 (+ 95us) server_negotiation.cc:407] Connection header received
0927 03:30:52.835199 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:52.835200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:52.835211 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:52.835257 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:30:52.835922 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:52.836415 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:52.837137 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:52.837312 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:52.837361 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:52.838526 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:30:52.838527 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:30:52.838528 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:30:52.838533 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:30:52.839951 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:52.840287 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:52.840288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:52.840288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:52.840335 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:30:52.840546 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:30:52.840547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:30:52.840547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:30:52.840684 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:30:52.840759 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:52.840854 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 03:30:52.840927 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 03:30:52.842631 15790 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:30:52.844328 15790 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:30:52.851138 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:52.847177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57758 (local address 127.0.98.2:38421)
0927 03:30:52.847339 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:52.847343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:52.847646 (+ 303us) server_negotiation.cc:407] Connection header received
0927 03:30:52.847740 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:52.847742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:52.847758 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:52.847840 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:30:52.847983 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:52.849030 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:52.849956 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:52.850695 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:52.850792 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:52.850852 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:52.851012 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 03:30:52.851048 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":98,"threads_started":1}
I20260927 03:30:52.851223 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:52.846750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38477 (local address 127.0.0.1:59276)
0927 03:30:52.847073 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:52.847088 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:52.847171 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:52.847470 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:52.847474 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:52.847476 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:30:52.847549 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:52.847550 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:52.848668 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:52.848671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:52.849912 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:52.849915 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:52.850935 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:52.850937 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:52.851005 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:52.851010 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:30:52.851044 (+ 34us) client_negotiation.cc:240] Negotiation successful
0927 03:30:52.851096 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":3,"thread_start_us":87,"threads_started":1}
I20260927 03:30:52.851157 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:52.846935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38421 (local address 127.0.0.1:57758)
0927 03:30:52.847580 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:52.847591 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:52.847653 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:52.847858 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:52.847859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:52.847862 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:30:52.847905 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:52.847907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:52.849110 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:52.849111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:52.849844 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:52.849846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:52.850795 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:52.850796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:52.850882 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:52.850897 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 03:30:52.850994 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 03:30:52.851048 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":72,"threads_started":1}
I20260927 03:30:52.851487 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:52.846864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59276 (local address 127.0.98.1:38477)
0927 03:30:52.847032 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:52.847035 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:52.847177 (+ 142us) server_negotiation.cc:407] Connection header received
0927 03:30:52.847339 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:52.847342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:52.847358 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:52.847469 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:30:52.847629 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:52.848514 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:52.850072 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:52.850821 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:52.851217 (+ 396us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:52.851290 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:52.851321 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 03:30:52.851370 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":70,"threads_started":1}
I20260927 03:30:52.851914 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:52.846720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33805 (local address 127.0.0.1:53776)
0927 03:30:52.847023 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:52.847067 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:52.847146 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:52.847623 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:52.847624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:52.847625 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:30:52.847665 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:52.847666 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:52.849039 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:52.849040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:52.850172 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:52.850174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:52.851121 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:52.851121 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:52.851709 (+ 588us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:52.851716 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 03:30:52.851799 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 03:30:52.851839 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":58,"thread_start_us":60,"threads_started":1}
I20260927 03:30:52.852104 16050 tablet_service.cc:1467] Processing CreateTablet for tablet 38be7df23f0c40f2903cd3d8019df2e0 (DEFAULT_TABLE table=test-table [id=6aed6433fcd742f888c09ffa79cf6c11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:30:52.852169 15916 tablet_service.cc:1467] Processing CreateTablet for tablet 38be7df23f0c40f2903cd3d8019df2e0 (DEFAULT_TABLE table=test-table [id=6aed6433fcd742f888c09ffa79cf6c11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:30:52.852459 16050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38be7df23f0c40f2903cd3d8019df2e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:30:52.852509 15916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38be7df23f0c40f2903cd3d8019df2e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:30:52.854988 16267 tablet_bootstrap.cc:492] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61: Bootstrap starting.
I20260927 03:30:52.855116 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:52.846818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53776 (local address 127.0.98.3:33805)
0927 03:30:52.847000 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:52.847005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:52.847281 (+ 276us) server_negotiation.cc:407] Connection header received
0927 03:30:52.847329 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:52.847331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:52.847349 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:52.847477 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:30:52.847747 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:52.848931 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:52.850273 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:52.850958 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:52.851003 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:52.851060 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:52.854979 (+ 3919us) server_negotiation.cc:299] Negotiation successful
0927 03:30:52.855037 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":96,"threads_started":1}
I20260927 03:30:52.855579 16268 tablet_bootstrap.cc:492] T 38be7df23f0c40f2903cd3d8019df2e0 P 33fa80feb50e4c3eb17927e5ccc7142f: Bootstrap starting.
I20260927 03:30:52.855805 16184 tablet_service.cc:1467] Processing CreateTablet for tablet 38be7df23f0c40f2903cd3d8019df2e0 (DEFAULT_TABLE table=test-table [id=6aed6433fcd742f888c09ffa79cf6c11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:30:52.856134 16184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38be7df23f0c40f2903cd3d8019df2e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:30:52.856213 16268 tablet_bootstrap.cc:654] T 38be7df23f0c40f2903cd3d8019df2e0 P 33fa80feb50e4c3eb17927e5ccc7142f: Neither blocks nor log segments found. Creating new log.
I20260927 03:30:52.856215 16267 tablet_bootstrap.cc:654] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61: Neither blocks nor log segments found. Creating new log.
I20260927 03:30:52.856683 16267 log.cc:824] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61: Log is configured to *not* fsync() on all Append() calls
I20260927 03:30:52.856788 16268 log.cc:824] T 38be7df23f0c40f2903cd3d8019df2e0 P 33fa80feb50e4c3eb17927e5ccc7142f: Log is configured to *not* fsync() on all Append() calls
I20260927 03:30:52.857434 16267 tablet_bootstrap.cc:492] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61: No bootstrap required, opened a new log
I20260927 03:30:52.857519 16267 ts_tablet_manager.cc:1397] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 03:30:52.858767 16271 tablet_bootstrap.cc:492] T 38be7df23f0c40f2903cd3d8019df2e0 P cb8b2d9f64ae44a0a6c65087194bf770: Bootstrap starting.
I20260927 03:30:52.859050 16267 raft_consensus.cc:348] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33805 } } peers { permanent_uuid: "b77d4b143c8d4406881fc7c1654dbf61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38477 } } peers { permanent_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38421 } }
I20260927 03:30:52.859223 16267 raft_consensus.cc:374] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:30:52.859273 16267 raft_consensus.cc:729] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b77d4b143c8d4406881fc7c1654dbf61, State: Initialized, Role: FOLLOWER
I20260927 03:30:52.859405 16271 tablet_bootstrap.cc:654] T 38be7df23f0c40f2903cd3d8019df2e0 P cb8b2d9f64ae44a0a6c65087194bf770: Neither blocks nor log segments found. Creating new log.
I20260927 03:30:52.859412 16267 consensus_queue.cc:260] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33805 } } peers { permanent_uuid: "b77d4b143c8d4406881fc7c1654dbf61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38477 } } peers { permanent_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38421 } }
I20260927 03:30:52.859771 16267 ts_tablet_manager.cc:1428] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 03:30:52.859783 15982 heartbeater.cc:503] Master 127.0.98.62:35465 was elected leader, sending a full tablet report...
I20260927 03:30:52.859939 16271 log.cc:824] T 38be7df23f0c40f2903cd3d8019df2e0 P cb8b2d9f64ae44a0a6c65087194bf770: Log is configured to *not* fsync() on all Append() calls
I20260927 03:30:52.860374 16268 tablet_bootstrap.cc:492] T 38be7df23f0c40f2903cd3d8019df2e0 P 33fa80feb50e4c3eb17927e5ccc7142f: No bootstrap required, opened a new log
I20260927 03:30:52.860438 16268 ts_tablet_manager.cc:1397] T 38be7df23f0c40f2903cd3d8019df2e0 P 33fa80feb50e4c3eb17927e5ccc7142f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 03:30:52.860834 16271 tablet_bootstrap.cc:492] T 38be7df23f0c40f2903cd3d8019df2e0 P cb8b2d9f64ae44a0a6c65087194bf770: No bootstrap required, opened a new log
I20260927 03:30:52.860899 16271 ts_tablet_manager.cc:1397] T 38be7df23f0c40f2903cd3d8019df2e0 P cb8b2d9f64ae44a0a6c65087194bf770: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:30:52.862092 16268 raft_consensus.cc:348] T 38be7df23f0c40f2903cd3d8019df2e0 P 33fa80feb50e4c3eb17927e5ccc7142f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33805 } } peers { permanent_uuid: "b77d4b143c8d4406881fc7c1654dbf61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38477 } } peers { permanent_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38421 } }
I20260927 03:30:52.862234 16268 raft_consensus.cc:374] T 38be7df23f0c40f2903cd3d8019df2e0 P 33fa80feb50e4c3eb17927e5ccc7142f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:30:52.862258 16268 raft_consensus.cc:729] T 38be7df23f0c40f2903cd3d8019df2e0 P 33fa80feb50e4c3eb17927e5ccc7142f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33fa80feb50e4c3eb17927e5ccc7142f, State: Initialized, Role: FOLLOWER
I20260927 03:30:52.862346 16268 consensus_queue.cc:260] T 38be7df23f0c40f2903cd3d8019df2e0 P 33fa80feb50e4c3eb17927e5ccc7142f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33805 } } peers { permanent_uuid: "b77d4b143c8d4406881fc7c1654dbf61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38477 } } peers { permanent_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38421 } }
I20260927 03:30:52.862267 16271 raft_consensus.cc:348] T 38be7df23f0c40f2903cd3d8019df2e0 P cb8b2d9f64ae44a0a6c65087194bf770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33805 } } peers { permanent_uuid: "b77d4b143c8d4406881fc7c1654dbf61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38477 } } peers { permanent_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38421 } }
I20260927 03:30:52.862429 16271 raft_consensus.cc:374] T 38be7df23f0c40f2903cd3d8019df2e0 P cb8b2d9f64ae44a0a6c65087194bf770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:30:52.862486 16271 raft_consensus.cc:729] T 38be7df23f0c40f2903cd3d8019df2e0 P cb8b2d9f64ae44a0a6c65087194bf770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb8b2d9f64ae44a0a6c65087194bf770, State: Initialized, Role: FOLLOWER
I20260927 03:30:52.862555 16268 ts_tablet_manager.cc:1428] T 38be7df23f0c40f2903cd3d8019df2e0 P 33fa80feb50e4c3eb17927e5ccc7142f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:30:52.862607 16116 heartbeater.cc:503] Master 127.0.98.62:35465 was elected leader, sending a full tablet report...
I20260927 03:30:52.862600 16271 consensus_queue.cc:260] T 38be7df23f0c40f2903cd3d8019df2e0 P cb8b2d9f64ae44a0a6c65087194bf770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33805 } } peers { permanent_uuid: "b77d4b143c8d4406881fc7c1654dbf61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38477 } } peers { permanent_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38421 } }
I20260927 03:30:52.862860 16271 ts_tablet_manager.cc:1428] T 38be7df23f0c40f2903cd3d8019df2e0 P cb8b2d9f64ae44a0a6c65087194bf770: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:30:52.862937 16250 heartbeater.cc:503] Master 127.0.98.62:35465 was elected leader, sending a full tablet report...
W20260927 03:30:52.912022 16117 tablet.cc:2369] T 38be7df23f0c40f2903cd3d8019df2e0 P 33fa80feb50e4c3eb17927e5ccc7142f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:30:52.995085 15983 tablet.cc:2369] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:30:53.064410 16251 tablet.cc:2369] T 38be7df23f0c40f2903cd3d8019df2e0 P cb8b2d9f64ae44a0a6c65087194bf770: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:30:53.179811 16272 raft_consensus.cc:482] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:30:53.179963 16272 raft_consensus.cc:504] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33805 } } peers { permanent_uuid: "b77d4b143c8d4406881fc7c1654dbf61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38477 } } peers { permanent_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38421 } }
I20260927 03:30:53.180261 16272 leader_election.cc:288] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb8b2d9f64ae44a0a6c65087194bf770 (127.0.98.3:33805), 33fa80feb50e4c3eb17927e5ccc7142f (127.0.98.2:38421)
I20260927 03:30:53.184544 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:53.180854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47397 (local address 127.0.98.2:38421)
0927 03:30:53.180906 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:53.180907 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:53.180939 (+ 32us) server_negotiation.cc:407] Connection header received
0927 03:30:53.181021 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:53.181022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:53.181033 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:53.181090 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:30:53.181250 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:53.181974 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:53.183604 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:53.184166 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:53.184242 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:53.184251 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:53.184432 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 03:30:53.184471 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 03:30:53.184544 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:53.180647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57395 (local address 127.0.98.3:33805)
0927 03:30:53.180723 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:53.180726 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:53.180811 (+ 85us) server_negotiation.cc:407] Connection header received
0927 03:30:53.180924 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:53.180925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:53.180939 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:53.181007 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:30:53.181198 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:53.181969 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:53.183603 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:53.184180 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:53.184242 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:53.184251 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:53.184432 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 03:30:53.184471 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 03:30:53.184579 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:53.180539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33805 (local address 127.0.98.1:57395)
0927 03:30:53.180713 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:53.180727 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:53.180829 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:53.181020 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:53.181021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:53.181023 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:30:53.181122 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:53.181124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:53.182057 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:53.182058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:53.183515 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:53.183518 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:53.184254 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:53.184255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:53.184345 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:53.184351 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 03:30:53.184413 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 03:30:53.184467 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":55,"thread_start_us":96,"threads_started":1}
I20260927 03:30:53.184582 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:53.180768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38421 (local address 127.0.98.1:47397)
0927 03:30:53.180884 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:53.180895 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:30:53.180951 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:30:53.181108 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:30:53.181108 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:30:53.181109 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:30:53.181177 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:53.181179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:53.182054 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:53.182054 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:53.183515 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:30:53.183518 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:30:53.184255 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:30:53.184256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:30:53.184345 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:53.184350 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:30:53.184413 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 03:30:53.184467 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":124,"thread_start_us":47,"threads_started":1}
I20260927 03:30:53.184989 16070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38be7df23f0c40f2903cd3d8019df2e0" candidate_uuid: "b77d4b143c8d4406881fc7c1654dbf61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" is_pre_election: true
I20260927 03:30:53.184994 16204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38be7df23f0c40f2903cd3d8019df2e0" candidate_uuid: "b77d4b143c8d4406881fc7c1654dbf61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" is_pre_election: true
I20260927 03:30:53.185149 16204 raft_consensus.cc:2454] T 38be7df23f0c40f2903cd3d8019df2e0 P cb8b2d9f64ae44a0a6c65087194bf770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b77d4b143c8d4406881fc7c1654dbf61 in term 0.
I20260927 03:30:53.185149 16070 raft_consensus.cc:2454] T 38be7df23f0c40f2903cd3d8019df2e0 P 33fa80feb50e4c3eb17927e5ccc7142f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b77d4b143c8d4406881fc7c1654dbf61 in term 0.
I20260927 03:30:53.185395 15869 leader_election.cc:302] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33fa80feb50e4c3eb17927e5ccc7142f, b77d4b143c8d4406881fc7c1654dbf61; no voters:
I20260927 03:30:53.185518 16272 raft_consensus.cc:2781] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:30:53.185572 16272 raft_consensus.cc:482] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:30:53.185597 16272 raft_consensus.cc:3037] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:30:53.186278 16272 raft_consensus.cc:504] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33805 } } peers { permanent_uuid: "b77d4b143c8d4406881fc7c1654dbf61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38477 } } peers { permanent_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38421 } }
I20260927 03:30:53.186417 16272 leader_election.cc:288] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61 [CANDIDATE]: Term 1 election: Requested vote from peers cb8b2d9f64ae44a0a6c65087194bf770 (127.0.98.3:33805), 33fa80feb50e4c3eb17927e5ccc7142f (127.0.98.2:38421)
I20260927 03:30:53.186717 16204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38be7df23f0c40f2903cd3d8019df2e0" candidate_uuid: "b77d4b143c8d4406881fc7c1654dbf61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb8b2d9f64ae44a0a6c65087194bf770"
I20260927 03:30:53.186729 16070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38be7df23f0c40f2903cd3d8019df2e0" candidate_uuid: "b77d4b143c8d4406881fc7c1654dbf61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33fa80feb50e4c3eb17927e5ccc7142f"
I20260927 03:30:53.186801 16070 raft_consensus.cc:3037] T 38be7df23f0c40f2903cd3d8019df2e0 P 33fa80feb50e4c3eb17927e5ccc7142f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:30:53.186801 16204 raft_consensus.cc:3037] T 38be7df23f0c40f2903cd3d8019df2e0 P cb8b2d9f64ae44a0a6c65087194bf770 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:30:53.187726 16204 raft_consensus.cc:2454] T 38be7df23f0c40f2903cd3d8019df2e0 P cb8b2d9f64ae44a0a6c65087194bf770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b77d4b143c8d4406881fc7c1654dbf61 in term 1.
I20260927 03:30:53.187814 16070 raft_consensus.cc:2454] T 38be7df23f0c40f2903cd3d8019df2e0 P 33fa80feb50e4c3eb17927e5ccc7142f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b77d4b143c8d4406881fc7c1654dbf61 in term 1.
I20260927 03:30:53.187981 15869 leader_election.cc:302] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33fa80feb50e4c3eb17927e5ccc7142f, b77d4b143c8d4406881fc7c1654dbf61; no voters:
I20260927 03:30:53.188131 16272 raft_consensus.cc:2781] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:30:53.188284 16272 raft_consensus.cc:686] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61 [term 1 LEADER]: Becoming Leader. State: Replica: b77d4b143c8d4406881fc7c1654dbf61, State: Running, Role: LEADER
I20260927 03:30:53.188405 16272 consensus_queue.cc:237] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33805 } } peers { permanent_uuid: "b77d4b143c8d4406881fc7c1654dbf61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38477 } } peers { permanent_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38421 } }
I20260927 03:30:53.189270 15789 catalog_manager.cc:5495] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61 reported cstate change: term changed from 0 to 1, leader changed from <none> to b77d4b143c8d4406881fc7c1654dbf61 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b77d4b143c8d4406881fc7c1654dbf61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b77d4b143c8d4406881fc7c1654dbf61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38421 } health_report { overall_health: UNKNOWN } } }
I20260927 03:30:53.205081 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:53.202724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59290 (local address 127.0.98.1:38477)
0927 03:30:53.202788 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:30:53.202790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:30:53.202828 (+ 38us) server_negotiation.cc:407] Connection header received
0927 03:30:53.202896 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:30:53.202897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:30:53.202914 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:30:53.202975 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:30:53.203103 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:53.203606 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:53.204366 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:30:53.204546 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:30:53.204599 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:30:53.204811 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:30:53.204913 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:30:53.204971 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 03:30:53.205010 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 03:30:53.205025 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 03:30:53.206787 16070 raft_consensus.cc:1261] T 38be7df23f0c40f2903cd3d8019df2e0 P 33fa80feb50e4c3eb17927e5ccc7142f [term 1 FOLLOWER]: Refusing update from remote peer b77d4b143c8d4406881fc7c1654dbf61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:30:53.206821 16204 raft_consensus.cc:1261] T 38be7df23f0c40f2903cd3d8019df2e0 P cb8b2d9f64ae44a0a6c65087194bf770 [term 1 FOLLOWER]: Refusing update from remote peer b77d4b143c8d4406881fc7c1654dbf61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:30:53.207151 16278 consensus_queue.cc:1035] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:30:53.207286 16272 consensus_queue.cc:1035] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:30:53.209911 16282 mvcc.cc:205] Tried to move back new op lower bound from 7333805478732300288 to 7333805478660243456. Current Snapshot: MvccSnapshot[applied={T|T < 7333805478732300288}]
I20260927 03:30:53.210131 16281 mvcc.cc:205] Tried to move back new op lower bound from 7333805478732300288 to 7333805478660243456. Current Snapshot: MvccSnapshot[applied={T|T < 7333805478732300288 or (T in {7333805478732300288})}]
I20260927 03:30:53.211606 15789 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:30:53.211705 15789 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:30:53.212387 15789 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327: Sending DeleteTablet for 3 replicas of tablet 38be7df23f0c40f2903cd3d8019df2e0
I20260927 03:30:53.212709 16050 tablet_service.cc:1514] Processing DeleteTablet for tablet 38be7df23f0c40f2903cd3d8019df2e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:30:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57758
I20260927 03:30:53.212723 16184 tablet_service.cc:1514] Processing DeleteTablet for tablet 38be7df23f0c40f2903cd3d8019df2e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:30:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53776
I20260927 03:30:53.212818 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 15853
I20260927 03:30:53.215540 16292 tablet_replica.cc:330] T 38be7df23f0c40f2903cd3d8019df2e0 P cb8b2d9f64ae44a0a6c65087194bf770: stopping tablet replica
I20260927 03:30:53.215646 16292 raft_consensus.cc:2229] T 38be7df23f0c40f2903cd3d8019df2e0 P cb8b2d9f64ae44a0a6c65087194bf770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:30:53.215765 16292 raft_consensus.cc:2258] T 38be7df23f0c40f2903cd3d8019df2e0 P cb8b2d9f64ae44a0a6c65087194bf770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:30:53.215940 16291 tablet_replica.cc:330] T 38be7df23f0c40f2903cd3d8019df2e0 P 33fa80feb50e4c3eb17927e5ccc7142f: stopping tablet replica
I20260927 03:30:53.216034 16291 raft_consensus.cc:2229] T 38be7df23f0c40f2903cd3d8019df2e0 P 33fa80feb50e4c3eb17927e5ccc7142f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:30:53.216144 16291 raft_consensus.cc:2258] T 38be7df23f0c40f2903cd3d8019df2e0 P 33fa80feb50e4c3eb17927e5ccc7142f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:30:53.216228 16292 ts_tablet_manager.cc:1905] T 38be7df23f0c40f2903cd3d8019df2e0 P cb8b2d9f64ae44a0a6c65087194bf770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:30:53.216552 16291 ts_tablet_manager.cc:1905] T 38be7df23f0c40f2903cd3d8019df2e0 P 33fa80feb50e4c3eb17927e5ccc7142f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:30:53.217958 16292 ts_tablet_manager.cc:1918] T 38be7df23f0c40f2903cd3d8019df2e0 P cb8b2d9f64ae44a0a6c65087194bf770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:30:53.218026 16292 log.cc:1196] T 38be7df23f0c40f2903cd3d8019df2e0 P cb8b2d9f64ae44a0a6c65087194bf770: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/wal/wals/38be7df23f0c40f2903cd3d8019df2e0
W20260927 03:30:53.218063 15776 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38477 (error 108)
I20260927 03:30:53.218145 16291 ts_tablet_manager.cc:1918] T 38be7df23f0c40f2903cd3d8019df2e0 P 33fa80feb50e4c3eb17927e5ccc7142f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:30:53.218207 16291 log.cc:1196] T 38be7df23f0c40f2903cd3d8019df2e0 P 33fa80feb50e4c3eb17927e5ccc7142f: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/wal/wals/38be7df23f0c40f2903cd3d8019df2e0
I20260927 03:30:53.218310 16292 ts_tablet_manager.cc:1939] T 38be7df23f0c40f2903cd3d8019df2e0 P cb8b2d9f64ae44a0a6c65087194bf770: Deleting consensus metadata
I20260927 03:30:53.218348 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 15987
I20260927 03:30:53.219314 15774 catalog_manager.cc:4841] TS cb8b2d9f64ae44a0a6c65087194bf770 (127.0.98.3:33805): tablet 38be7df23f0c40f2903cd3d8019df2e0 (table test-table [id=6aed6433fcd742f888c09ffa79cf6c11]) successfully deleted
I20260927 03:30:53.219806 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:53.219672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38477 (local address 127.0.0.1:59298)
0927 03:30:53.219733 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:53.219780 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260927 03:30:53.220003 15776 catalog_manager.cc:4584] TS b77d4b143c8d4406881fc7c1654dbf61 (127.0.98.1:38477): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 38be7df23f0c40f2903cd3d8019df2e0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38477: connect: Connection refused (error 111)
I20260927 03:30:53.223624 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 16121
I20260927 03:30:53.223657 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:30:53.223543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38421 (local address 127.0.0.1:57772)
0927 03:30:53.223597 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 03:30:53.223622 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 03:30:53.227895 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 15758
I20260927 03:31:13.233358 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35465
--webserver_interface=127.0.98.62
--webserver_port=36779
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:31:13.314682 16294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:13.314900 16294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:13.314925 16294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:13.316116 16294 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 03:31:13.316162 16294 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 03:31:13.316176 16294 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 03:31:13.316188 16294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:31:13.316236 16294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:13.316267 16294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:13.316284 16294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:31:13.316308 16294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:31:13.317778 16294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:13.318041 16294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:13.318287 16294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:31:13.326881 16301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:13.326890 16300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:13.326895 16303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:13.327464 16294 server_base.cc:1034] running on GCE node
I20260927 03:31:13.327725 16294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:13.328002 16294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:13.329164 16294 hybrid_clock.cc:648] HybridClock initialized: now 1790479873329147 us; error 30 us; skew 500 ppm
Sep 27 03:31:13 dist-test-slave-j0hf krb5kdc[15745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:31:13.331691 16294 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:31:13.332193 16294 webserver.cc:460] Webserver started at http://127.0.98.62:36779/ using document root <none> and password file <none>
I20260927 03:31:13.332438 16294 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:13.332515 16294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:13.334005 16294 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:31:13.334707 16310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:13.334975 16294 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:31:13.335085 16294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "ac032e00c98e4600a63fd0da0e315327"
format_stamp: "Formatted at 2026-09-27 03:30:52 on dist-test-slave-j0hf"
server_key: "5ee7528513c5b10ecf32bf3c3eea2548"
server_key_iv: "8abe229cc80596e3e7a28fdbc1e9bb3c"
server_key_version: "encryptionkey@0"
I20260927 03:31:13.335572 16294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:13.347882 16294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:13.350006 16294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:13.350173 16294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:13.354444 16294 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35465
I20260927 03:31:13.354476 16362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35465 every 8 connection(s)
I20260927 03:31:13.354907 16294 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:31:13.356415 16363 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:31:13.357111 16363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327: Bootstrap starting.
I20260927 03:31:13.359112 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 16294
I20260927 03:31:13.359475 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38477
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35465
--webserver_port=44781
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:31:13.365880 16363 log.cc:824] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327: Log is configured to *not* fsync() on all Append() calls
I20260927 03:31:13.367842 16363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:31:13.368135 16363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327: Bootstrap complete.
I20260927 03:31:13.370384 16363 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac032e00c98e4600a63fd0da0e315327" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35465 } }
I20260927 03:31:13.370602 16363 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac032e00c98e4600a63fd0da0e315327, State: Initialized, Role: FOLLOWER
I20260927 03:31:13.370774 16363 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac032e00c98e4600a63fd0da0e315327" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35465 } }
I20260927 03:31:13.370832 16363 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:31:13.370884 16363 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:31:13.370932 16363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [term 1 FOLLOWER]: Advancing to term 2
I20260927 03:31:13.371973 16363 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac032e00c98e4600a63fd0da0e315327" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35465 } }
I20260927 03:31:13.372083 16363 leader_election.cc:302] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac032e00c98e4600a63fd0da0e315327; no voters:
I20260927 03:31:13.372289 16363 leader_election.cc:288] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 03:31:13.372323 16367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 03:31:13.372438 16367 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [term 2 LEADER]: Becoming Leader. State: Replica: ac032e00c98e4600a63fd0da0e315327, State: Running, Role: LEADER
I20260927 03:31:13.372555 16363 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:31:13.372594 16367 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac032e00c98e4600a63fd0da0e315327" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35465 } }
I20260927 03:31:13.373036 16369 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac032e00c98e4600a63fd0da0e315327. Latest consensus state: current_term: 2 leader_uuid: "ac032e00c98e4600a63fd0da0e315327" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac032e00c98e4600a63fd0da0e315327" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35465 } } }
I20260927 03:31:13.373127 16369 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [sys.catalog]: This master's current role is: LEADER
I20260927 03:31:13.373265 16368 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ac032e00c98e4600a63fd0da0e315327" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac032e00c98e4600a63fd0da0e315327" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35465 } } }
I20260927 03:31:13.373311 16368 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [sys.catalog]: This master's current role is: LEADER
I20260927 03:31:13.373473 16375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:31:13.374404 16375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:31:13.374569 16375 catalog_manager.cc:1259] Loaded cluster ID: d2c2c459365d4fe2aeda94b99a865a1f
I20260927 03:31:13.374835 16375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:31:13.376044 16375 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:31:13.376243 16375 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327: Loaded TSK: 0
I20260927 03:31:13.386364 16375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327: Generated new TSK 1
I20260927 03:31:13.386484 16375 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 03:31:13.464442 16365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:13.464671 16365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:13.464711 16365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:13.465890 16365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:13.465965 16365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:13.466008 16365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:31:13.467600 16365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38477
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44781
--tserver_master_addrs=127.0.98.62:35465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:13.467878 16365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:13.468133 16365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:31:13.474378 16390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:13.474433 16391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:13.474393 16393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:13.475468 16365 server_base.cc:1034] running on GCE node
I20260927 03:31:13.475701 16365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:13.475962 16365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:13.477149 16365 hybrid_clock.cc:648] HybridClock initialized: now 1790479873477135 us; error 33 us; skew 500 ppm
Sep 27 03:31:13 dist-test-slave-j0hf krb5kdc[15745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:31:13.479907 16365 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:31:13.480438 16365 webserver.cc:460] Webserver started at http://127.0.98.1:44781/ using document root <none> and password file <none>
I20260927 03:31:13.480746 16365 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:13.480835 16365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:13.482703 16365 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:31:13.483522 16400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:13.483722 16365 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:31:13.483814 16365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "b77d4b143c8d4406881fc7c1654dbf61"
format_stamp: "Formatted at 2026-09-27 03:30:52 on dist-test-slave-j0hf"
server_key: "c90e9ad6f2a4c8b27674a814971a62df"
server_key_iv: "acc6c050ad6c1c9f35c598926bb92998"
server_key_version: "encryptionkey@0"
I20260927 03:31:13.484270 16365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:13.503857 16365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:13.506300 16365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:13.506454 16365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:13.506722 16365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:31:13.507210 16407 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 03:31:13.508759 16365 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 03:31:13.508827 16365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 03:31:13.508862 16365 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 03:31:13.509711 16365 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 03:31:13.509758 16365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 03:31:13.509824 16407 tablet_bootstrap.cc:492] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61: Bootstrap starting.
I20260927 03:31:13.516877 16365 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38477
I20260927 03:31:13.517108 16514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38477 every 8 connection(s)
I20260927 03:31:13.517398 16365 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:31:13.520483 16407 log.cc:824] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61: Log is configured to *not* fsync() on all Append() calls
I20260927 03:31:13.523233 16407 tablet_bootstrap.cc:492] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:31:13.523645 16407 tablet_bootstrap.cc:492] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61: Bootstrap complete.
I20260927 03:31:13.524907 16407 ts_tablet_manager.cc:1397] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s
I20260927 03:31:13.525240 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 16365
I20260927 03:31:13.525545 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38421
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35465
--webserver_port=33211
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 03:31:13 dist-test-slave-j0hf krb5kdc[15745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:31:13.526222 16407 raft_consensus.cc:348] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33805 } } peers { permanent_uuid: "b77d4b143c8d4406881fc7c1654dbf61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38477 } } peers { permanent_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38421 } }
I20260927 03:31:13.526505 16407 raft_consensus.cc:729] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b77d4b143c8d4406881fc7c1654dbf61, State: Initialized, Role: FOLLOWER
I20260927 03:31:13.526635 16407 consensus_queue.cc:260] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33805 } } peers { permanent_uuid: "b77d4b143c8d4406881fc7c1654dbf61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38477 } } peers { permanent_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38421 } }
I20260927 03:31:13.526877 16407 ts_tablet_manager.cc:1428] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 03:31:13.530604 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:13.518109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.1:42933)
0927 03:31:13.518314 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:13.518365 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:13.519179 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:13.520110 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:13.520115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:13.520491 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:31:13.520784 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:13.520792 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.522017 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.522018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:13.522815 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:13.522818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.523675 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.523676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:13.523870 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:13.524456 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:31:13.524465 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:31:13.526145 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:31:13.528732 (+ 2587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:31:13.528739 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:31:13.528741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:31:13.529034 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:31:13.529340 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:31:13.529341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:31:13.529342 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:31:13.529470 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:31:13.529933 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:31:13.529939 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:31:13.530093 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 03:31:13.530228 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 03:31:13.530391 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":102,"threads_started":1}
I20260927 03:31:13.530714 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:13.518207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42933 (local address 127.0.98.62:35465)
0927 03:31:13.519584 (+ 1377us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:13.519596 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:13.519649 (+ 53us) server_negotiation.cc:407] Connection header received
0927 03:31:13.519812 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:13.519821 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:13.519885 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:13.520050 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:13.520928 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.521872 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.522915 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.523550 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.523686 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:13.526263 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:13.526267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:13.526268 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:31:13.526279 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:13.528641 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:13.529128 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:13.529131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:13.529132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:13.529251 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:13.529540 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:13.529541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:13.529541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:13.529843 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:13.529921 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:13.530308 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 03:31:13.530465 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1317,"thread_start_us":93,"threads_started":1}
I20260927 03:31:13.531828 16515 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35465
I20260927 03:31:13.531936 16515 heartbeater.cc:464] Registering TS with master...
I20260927 03:31:13.532146 16515 heartbeater.cc:511] Master 127.0.98.62:35465 requested a full tablet report, sending...
I20260927 03:31:13.533013 16327 ts_manager.cc:194] Registered new tserver with Master: b77d4b143c8d4406881fc7c1654dbf61 (127.0.98.1:38477)
I20260927 03:31:13.534353 16327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42933
I20260927 03:31:13.535487 16515 heartbeater.cc:503] Master 127.0.98.62:35465 was elected leader, sending a full tablet report...
I20260927 03:31:13.547086 16522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:13.533707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38477 (local address 127.0.0.1:55648)
0927 03:31:13.543520 (+ 9813us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:13.543562 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:13.543662 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:13.543859 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:13.543862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:13.543865 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:13.543917 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:13.543919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.545183 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.545184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:13.545969 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:13.545971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.546858 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.546859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:13.546916 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:13.546926 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 03:31:13.546975 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 03:31:13.547013 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9779,"thread_start_us":85,"threads_started":1}
I20260927 03:31:13.547117 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:13.533965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55648 (local address 127.0.98.1:38477)
0927 03:31:13.534138 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:13.534143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:13.543659 (+ 9516us) server_negotiation.cc:407] Connection header received
0927 03:31:13.543759 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:13.543762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:13.543779 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:13.543851 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:13.543994 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.545052 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.546052 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.546773 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.546848 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:13.546902 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:13.546996 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 03:31:13.547030 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260927 03:31:13.547537 16449 tablet_service.cc:1514] Processing DeleteTablet for tablet 38be7df23f0c40f2903cd3d8019df2e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:30:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55648
I20260927 03:31:13.548041 16524 tablet_replica.cc:330] stopping tablet replica
I20260927 03:31:13.548158 16524 raft_consensus.cc:2229] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:31:13.548234 16524 raft_consensus.cc:2258] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:31:13.548553 16524 ts_tablet_manager.cc:1905] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:31:13.551151 16524 ts_tablet_manager.cc:1918] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:31:13.551227 16524 log.cc:1196] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/wal/wals/38be7df23f0c40f2903cd3d8019df2e0
I20260927 03:31:13.551477 16524 ts_tablet_manager.cc:1939] T 38be7df23f0c40f2903cd3d8019df2e0 P b77d4b143c8d4406881fc7c1654dbf61: Deleting consensus metadata
I20260927 03:31:13.552007 16313 catalog_manager.cc:4841] TS b77d4b143c8d4406881fc7c1654dbf61 (127.0.98.1:38477): tablet 38be7df23f0c40f2903cd3d8019df2e0 (table test-table [id=6aed6433fcd742f888c09ffa79cf6c11]) successfully deleted
W20260927 03:31:13.612727 16520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:13.612985 16520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:13.613023 16520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:13.614205 16520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:13.614280 16520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:13.614322 16520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:31:13.615800 16520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33211
--tserver_master_addrs=127.0.98.62:35465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:13.616070 16520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:13.616309 16520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:31:13.621388 16529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:13.621399 16530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:13.621619 16532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:13.621574 16520 server_base.cc:1034] running on GCE node
I20260927 03:31:13.621851 16520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:13.622118 16520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:13.623283 16520 hybrid_clock.cc:648] HybridClock initialized: now 1790479873623266 us; error 51 us; skew 500 ppm
Sep 27 03:31:13 dist-test-slave-j0hf krb5kdc[15745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:31:13.625789 16520 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:31:13.626241 16520 webserver.cc:460] Webserver started at http://127.0.98.2:33211/ using document root <none> and password file <none>
I20260927 03:31:13.626494 16520 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:13.626587 16520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:13.628041 16520 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:31:13.628798 16539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:13.628970 16520 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:31:13.629099 16520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "33fa80feb50e4c3eb17927e5ccc7142f"
format_stamp: "Formatted at 2026-09-27 03:30:52 on dist-test-slave-j0hf"
server_key: "1a539d321fc934b389ee8dd631a6767e"
server_key_iv: "f88e40753c245127a2457df7b6f29ef6"
server_key_version: "encryptionkey@0"
I20260927 03:31:13.629474 16520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:13.647069 16520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:13.649051 16520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:13.649214 16520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:13.649454 16520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:31:13.649875 16546 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260927 03:31:13.651274 16546 ts_tablet_manager.cc:1852] T 38be7df23f0c40f2903cd3d8019df2e0 P 33fa80feb50e4c3eb17927e5ccc7142f: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260927 03:31:13.651614 16546 ts_tablet_manager.cc:1905] T 38be7df23f0c40f2903cd3d8019df2e0 P 33fa80feb50e4c3eb17927e5ccc7142f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:31:13.654256 16546 ts_tablet_manager.cc:1918] T 38be7df23f0c40f2903cd3d8019df2e0 P 33fa80feb50e4c3eb17927e5ccc7142f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:31:13.654333 16546 log.cc:1196] T 38be7df23f0c40f2903cd3d8019df2e0 P 33fa80feb50e4c3eb17927e5ccc7142f: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/wal/wals/38be7df23f0c40f2903cd3d8019df2e0
I20260927 03:31:13.654520 16546 ts_tablet_manager.cc:1939] T 38be7df23f0c40f2903cd3d8019df2e0 P 33fa80feb50e4c3eb17927e5ccc7142f: Deleting consensus metadata
I20260927 03:31:13.654776 16520 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260927 03:31:13.654840 16520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 03:31:13.654886 16520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:31:13.654912 16520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:13.660678 16520 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38421
I20260927 03:31:13.660727 16653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38421 every 8 connection(s)
I20260927 03:31:13.661017 16520 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:31:13.662022 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 16520
I20260927 03:31:13.662271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33805
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35465
--webserver_port=42765
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 03:31:13 dist-test-slave-j0hf krb5kdc[15745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:31:13.670883 16656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:13.661696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.2:36835)
0927 03:31:13.661873 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:13.661914 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:13.662591 (+ 677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:13.662903 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:13.662907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:13.663173 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:31:13.663366 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:13.663370 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.664150 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.664151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:13.664985 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:13.664987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.665380 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.665381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:13.665526 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:13.666095 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:31:13.666104 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:31:13.667601 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:31:13.669279 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:31:13.669286 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:31:13.669288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:31:13.669598 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:31:13.669860 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:31:13.669861 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:31:13.669861 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:31:13.669975 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:31:13.670311 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:31:13.670316 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:31:13.670490 (+ 174us) client_negotiation.cc:769] Sending connection context
0927 03:31:13.670579 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 03:31:13.670713 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":85,"threads_started":1}
I20260927 03:31:13.670890 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:13.661694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36835 (local address 127.0.98.62:35465)
0927 03:31:13.661737 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:13.661739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:13.662004 (+ 265us) server_negotiation.cc:407] Connection header received
0927 03:31:13.662700 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:13.662701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:13.662726 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:13.662823 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:13.663520 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.664046 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.665080 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.665285 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.665373 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:13.667698 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:13.667699 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:13.667700 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:31:13.667708 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:13.669178 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:13.669693 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:13.669695 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:13.669695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:13.669778 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:13.670062 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:13.670063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:13.670063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:13.670234 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:13.670318 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:13.670632 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 03:31:13.670751 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:31:13.671471 16654 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35465
I20260927 03:31:13.671590 16654 heartbeater.cc:464] Registering TS with master...
I20260927 03:31:13.671805 16654 heartbeater.cc:511] Master 127.0.98.62:35465 requested a full tablet report, sending...
I20260927 03:31:13.672252 16327 ts_manager.cc:194] Registered new tserver with Master: 33fa80feb50e4c3eb17927e5ccc7142f (127.0.98.2:38421)
I20260927 03:31:13.672787 16327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36835
W20260927 03:31:13.747601 16657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:13.747807 16657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:13.747830 16657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:13.749001 16657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:13.749044 16657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:13.749073 16657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:31:13.750545 16657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33805
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42765
--tserver_master_addrs=127.0.98.62:35465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:13.750801 16657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:13.751029 16657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:31:13.755955 16665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:13.755977 16663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:13.755927 16662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:13.756124 16657 server_base.cc:1034] running on GCE node
I20260927 03:31:13.756333 16657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:13.756596 16657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:13.757761 16657 hybrid_clock.cc:648] HybridClock initialized: now 1790479873757757 us; error 40 us; skew 500 ppm
Sep 27 03:31:13 dist-test-slave-j0hf krb5kdc[15745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:31:13.760401 16657 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:31:13.760878 16657 webserver.cc:460] Webserver started at http://127.0.98.3:42765/ using document root <none> and password file <none>
I20260927 03:31:13.761121 16657 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:13.761202 16657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:13.762537 16657 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:31:13.763145 16672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:13.763316 16657 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:31:13.763425 16657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "cb8b2d9f64ae44a0a6c65087194bf770"
format_stamp: "Formatted at 2026-09-27 03:30:52 on dist-test-slave-j0hf"
server_key: "96524ede5ba89976c92496a281b35074"
server_key_iv: "3998a782d2aba15a960a0a4bd54ffede"
server_key_version: "encryptionkey@0"
I20260927 03:31:13.763837 16657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:13.780112 16657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:13.782635 16657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:13.782804 16657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:13.783047 16657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:31:13.783354 16657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:31:13.783406 16657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:13.783484 16657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:31:13.783535 16657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:13.789618 16657 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33805
I20260927 03:31:13.789669 16785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33805 every 8 connection(s)
I20260927 03:31:13.790001 16657 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:31:13 dist-test-slave-j0hf krb5kdc[15745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:31:13.797394 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 16657
I20260927 03:31:13.799028 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:13.790726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54891 (local address 127.0.98.62:35465)
0927 03:31:13.790772 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:13.790774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:13.791008 (+ 234us) server_negotiation.cc:407] Connection header received
0927 03:31:13.791490 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:13.791492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:13.791503 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:13.791555 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:13.792431 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.792919 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.793867 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.794050 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.794104 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:13.796298 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:13.796300 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:13.796300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:31:13.796307 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:13.797546 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:13.797988 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:13.797989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:13.797990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:13.798056 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:13.798362 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:13.798363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:13.798363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:13.798517 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:13.798593 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:13.798836 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 03:31:13.798923 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:31:13.799127 16788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:13.790733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.3:54891)
0927 03:31:13.790885 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:13.790919 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:13.791382 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:13.791610 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:13.791613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:13.792160 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:31:13.792352 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:13.792356 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.793001 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.793001 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:13.793793 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:13.793795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.794133 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.794134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:13.794235 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:13.794805 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:31:13.794814 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:31:13.796211 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:31:13.797647 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:31:13.797653 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:31:13.797655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:31:13.797918 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:31:13.798133 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:31:13.798133 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:31:13.798134 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:31:13.798279 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:31:13.798585 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:31:13.798590 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:31:13.798704 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 03:31:13.798784 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 03:31:13.798897 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":91,"threads_started":1}
I20260927 03:31:13.799772 16786 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35465
I20260927 03:31:13.799853 16786 heartbeater.cc:464] Registering TS with master...
I20260927 03:31:13.800043 16786 heartbeater.cc:511] Master 127.0.98.62:35465 requested a full tablet report, sending...
I20260927 03:31:13.800438 16327 ts_manager.cc:194] Registered new tserver with Master: cb8b2d9f64ae44a0a6c65087194bf770 (127.0.98.3:33805)
I20260927 03:31:13.800822 16327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54891
I20260927 03:31:13.804443 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:13.797842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42974 (local address 127.0.98.62:35465)
0927 03:31:13.797968 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:13.797970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:13.798061 (+ 91us) server_negotiation.cc:407] Connection header received
0927 03:31:13.798130 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:13.798131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:13.798143 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:13.798215 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:13.798970 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.799607 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.800590 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.800815 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.800898 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:13.802148 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:13.802149 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:13.802149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:31:13.802156 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:13.803317 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:13.803683 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:13.803684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:13.803684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:13.803745 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:13.803950 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:13.803951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:13.803951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:13.804056 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:13.804234 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:13.804269 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 03:31:13.804342 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260927 03:31:13.805155 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:31:13.807708 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:13.805513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42988 (local address 127.0.98.62:35465)
0927 03:31:13.805544 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:13.805546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:13.805654 (+ 108us) server_negotiation.cc:407] Connection header received
0927 03:31:13.805698 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:13.805699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:13.805725 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:13.805784 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:31:13.805852 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.806283 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.807035 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.807252 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.807317 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:13.807490 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:31:13.807544 (+ 54us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 03:31:13.807666 (+ 122us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42988: token expired
Metrics: {"server-negotiator.queue_time_us":15}
W20260927 03:31:13.807777 16790 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42988: token expired
W20260927 03:31:13.808756 16253 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:35465)
I20260927 03:31:13.815809 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:13.808993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43004 (local address 127.0.98.62:35465)
0927 03:31:13.809029 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:13.809031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:13.809040 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:31:13.809063 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:13.809064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:13.809074 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:13.809143 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:13.809575 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.810330 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.811191 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.811380 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.811511 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:13.812770 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:13.812771 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:13.812772 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:31:13.812780 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:13.814490 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:13.814880 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:13.814882 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:13.814882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:13.814950 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:13.815169 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:13.815170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:13.815170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:13.815347 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:13.815533 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:13.815609 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 03:31:13.815696 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:31:13.817041 16327 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:31:13.818639 16327 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:31:13.821111 16449 tablet_service.cc:1467] Processing CreateTablet for tablet 1397066a6b98460791300c435af57363 (DEFAULT_TABLE table=test-table [id=590a61959fa74f08931e6e5d5888b2e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:31:13.821314 16449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1397066a6b98460791300c435af57363. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:31:13.822774 16407 tablet_bootstrap.cc:492] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61: Bootstrap starting.
I20260927 03:31:13.823128 16407 tablet_bootstrap.cc:654] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61: Neither blocks nor log segments found. Creating new log.
I20260927 03:31:13.824779 16407 tablet_bootstrap.cc:492] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61: No bootstrap required, opened a new log
I20260927 03:31:13.824841 16407 ts_tablet_manager.cc:1397] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260927 03:31:13.824903 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:13.821018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38421 (local address 127.0.0.1:33388)
0927 03:31:13.821183 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:13.821194 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:13.821240 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:13.821697 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:13.821698 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:13.821701 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:13.821753 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:13.821755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.822780 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.822781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:13.823700 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:13.823702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.824665 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.824667 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:13.824738 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:13.824744 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 03:31:13.824777 (+ 33us) client_negotiation.cc:240] Negotiation successful
0927 03:31:13.824828 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":94,"threads_started":1}
I20260927 03:31:13.824970 16407 raft_consensus.cc:348] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33805 } } peers { permanent_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38421 } } peers { permanent_uuid: "b77d4b143c8d4406881fc7c1654dbf61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38477 } }
I20260927 03:31:13.825066 16407 raft_consensus.cc:374] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:31:13.825119 16407 raft_consensus.cc:729] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b77d4b143c8d4406881fc7c1654dbf61, State: Initialized, Role: FOLLOWER
I20260927 03:31:13.825176 16407 consensus_queue.cc:260] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33805 } } peers { permanent_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38421 } } peers { permanent_uuid: "b77d4b143c8d4406881fc7c1654dbf61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38477 } }
I20260927 03:31:13.825276 16407 ts_tablet_manager.cc:1428] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:13.825400 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:13.821275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33388 (local address 127.0.98.2:38421)
0927 03:31:13.821501 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:13.821505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:13.821514 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:31:13.821550 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:13.821551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:13.821564 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:13.821673 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:13.821832 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.822689 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.823789 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.824547 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.825158 (+ 611us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:13.825243 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:13.825271 (+ 28us) server_negotiation.cc:299] Negotiation successful
0927 03:31:13.825316 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":110,"threads_started":1}
I20260927 03:31:13.825992 16588 tablet_service.cc:1467] Processing CreateTablet for tablet 1397066a6b98460791300c435af57363 (DEFAULT_TABLE table=test-table [id=590a61959fa74f08931e6e5d5888b2e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:31:13.826391 16588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1397066a6b98460791300c435af57363. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:31:13.826650 16522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:13.820909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33805 (local address 127.0.0.1:45090)
0927 03:31:13.820948 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:13.820960 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:13.820996 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:13.823313 (+ 2317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:13.823314 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:13.823316 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:13.823358 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:13.823359 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.824367 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.824368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:13.825430 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:13.825434 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.826398 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.826399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:13.826464 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:13.826469 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:31:13.826527 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 03:31:13.826587 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260927 03:31:13.826664 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:13.821028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45090 (local address 127.0.98.3:33805)
0927 03:31:13.823111 (+ 2083us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:13.823117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:13.823130 (+ 13us) server_negotiation.cc:407] Connection header received
0927 03:31:13.823171 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:13.823173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:13.823190 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:13.823303 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:13.823411 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.824289 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.825525 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.826291 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.826395 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:13.826456 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:13.826540 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 03:31:13.826580 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2037,"thread_start_us":120,"threads_started":1}
I20260927 03:31:13.827308 16720 tablet_service.cc:1467] Processing CreateTablet for tablet 1397066a6b98460791300c435af57363 (DEFAULT_TABLE table=test-table [id=590a61959fa74f08931e6e5d5888b2e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:31:13.827656 16720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1397066a6b98460791300c435af57363. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:31:13.828533 16546 tablet_bootstrap.cc:492] T 1397066a6b98460791300c435af57363 P 33fa80feb50e4c3eb17927e5ccc7142f: Bootstrap starting.
I20260927 03:31:13.829257 16546 tablet_bootstrap.cc:654] T 1397066a6b98460791300c435af57363 P 33fa80feb50e4c3eb17927e5ccc7142f: Neither blocks nor log segments found. Creating new log.
I20260927 03:31:13.829497 16546 log.cc:824] T 1397066a6b98460791300c435af57363 P 33fa80feb50e4c3eb17927e5ccc7142f: Log is configured to *not* fsync() on all Append() calls
I20260927 03:31:13.830142 16546 tablet_bootstrap.cc:492] T 1397066a6b98460791300c435af57363 P 33fa80feb50e4c3eb17927e5ccc7142f: No bootstrap required, opened a new log
I20260927 03:31:13.830188 16546 ts_tablet_manager.cc:1397] T 1397066a6b98460791300c435af57363 P 33fa80feb50e4c3eb17927e5ccc7142f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:31:13.830492 16797 tablet_bootstrap.cc:492] T 1397066a6b98460791300c435af57363 P cb8b2d9f64ae44a0a6c65087194bf770: Bootstrap starting.
I20260927 03:31:13.831036 16797 tablet_bootstrap.cc:654] T 1397066a6b98460791300c435af57363 P cb8b2d9f64ae44a0a6c65087194bf770: Neither blocks nor log segments found. Creating new log.
I20260927 03:31:13.831622 16797 log.cc:824] T 1397066a6b98460791300c435af57363 P cb8b2d9f64ae44a0a6c65087194bf770: Log is configured to *not* fsync() on all Append() calls
I20260927 03:31:13.831765 16546 raft_consensus.cc:348] T 1397066a6b98460791300c435af57363 P 33fa80feb50e4c3eb17927e5ccc7142f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33805 } } peers { permanent_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38421 } } peers { permanent_uuid: "b77d4b143c8d4406881fc7c1654dbf61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38477 } }
I20260927 03:31:13.831939 16546 raft_consensus.cc:374] T 1397066a6b98460791300c435af57363 P 33fa80feb50e4c3eb17927e5ccc7142f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:31:13.831966 16546 raft_consensus.cc:729] T 1397066a6b98460791300c435af57363 P 33fa80feb50e4c3eb17927e5ccc7142f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33fa80feb50e4c3eb17927e5ccc7142f, State: Initialized, Role: FOLLOWER
I20260927 03:31:13.832120 16546 consensus_queue.cc:260] T 1397066a6b98460791300c435af57363 P 33fa80feb50e4c3eb17927e5ccc7142f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33805 } } peers { permanent_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38421 } } peers { permanent_uuid: "b77d4b143c8d4406881fc7c1654dbf61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38477 } }
I20260927 03:31:13.832424 16546 ts_tablet_manager.cc:1428] T 1397066a6b98460791300c435af57363 P 33fa80feb50e4c3eb17927e5ccc7142f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 03:31:13.832459 16797 tablet_bootstrap.cc:492] T 1397066a6b98460791300c435af57363 P cb8b2d9f64ae44a0a6c65087194bf770: No bootstrap required, opened a new log
I20260927 03:31:13.832430 16654 heartbeater.cc:503] Master 127.0.98.62:35465 was elected leader, sending a full tablet report...
I20260927 03:31:13.832548 16797 ts_tablet_manager.cc:1397] T 1397066a6b98460791300c435af57363 P cb8b2d9f64ae44a0a6c65087194bf770: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:31:13.834411 16797 raft_consensus.cc:348] T 1397066a6b98460791300c435af57363 P cb8b2d9f64ae44a0a6c65087194bf770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33805 } } peers { permanent_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38421 } } peers { permanent_uuid: "b77d4b143c8d4406881fc7c1654dbf61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38477 } }
I20260927 03:31:13.834571 16797 raft_consensus.cc:374] T 1397066a6b98460791300c435af57363 P cb8b2d9f64ae44a0a6c65087194bf770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:31:13.834609 16797 raft_consensus.cc:729] T 1397066a6b98460791300c435af57363 P cb8b2d9f64ae44a0a6c65087194bf770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb8b2d9f64ae44a0a6c65087194bf770, State: Initialized, Role: FOLLOWER
I20260927 03:31:13.834721 16797 consensus_queue.cc:260] T 1397066a6b98460791300c435af57363 P cb8b2d9f64ae44a0a6c65087194bf770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33805 } } peers { permanent_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38421 } } peers { permanent_uuid: "b77d4b143c8d4406881fc7c1654dbf61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38477 } }
I20260927 03:31:13.834998 16797 ts_tablet_manager.cc:1428] T 1397066a6b98460791300c435af57363 P cb8b2d9f64ae44a0a6c65087194bf770: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 03:31:13.835033 16786 heartbeater.cc:503] Master 127.0.98.62:35465 was elected leader, sending a full tablet report...
I20260927 03:31:13.850544 16521 raft_consensus.cc:482] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:31:13.850687 16521 raft_consensus.cc:504] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33805 } } peers { permanent_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38421 } } peers { permanent_uuid: "b77d4b143c8d4406881fc7c1654dbf61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38477 } }
I20260927 03:31:13.851107 16521 leader_election.cc:288] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb8b2d9f64ae44a0a6c65087194bf770 (127.0.98.3:33805), 33fa80feb50e4c3eb17927e5ccc7142f (127.0.98.2:38421)
I20260927 03:31:13.854354 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:13.851377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48035 (local address 127.0.98.3:33805)
0927 03:31:13.851466 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:13.851468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:13.851507 (+ 39us) server_negotiation.cc:407] Connection header received
0927 03:31:13.851539 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:13.851540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:13.851553 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:13.851606 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:13.852166 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.852634 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.853604 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.853983 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.854079 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:13.854090 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:13.854245 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 03:31:13.854287 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 03:31:13.854378 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:13.851287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33805 (local address 127.0.98.1:48035)
0927 03:31:13.851360 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:13.851373 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:13.851446 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:13.851662 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:13.851663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:13.851665 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:13.852096 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:13.852098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.852719 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.852720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:13.853503 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:13.853505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.854080 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.854080 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:13.854171 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:13.854174 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:31:13.854238 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 03:31:13.854282 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 03:31:13.854380 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:13.851593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32973 (local address 127.0.98.2:38421)
0927 03:31:13.851634 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:13.851636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:13.851773 (+ 137us) server_negotiation.cc:407] Connection header received
0927 03:31:13.851829 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:13.851830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:13.851843 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:13.851919 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:13.852113 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.852596 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.853610 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.853998 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.854079 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:13.854091 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:13.854270 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 03:31:13.854311 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:31:13.854674 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:13.851419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38421 (local address 127.0.98.1:32973)
0927 03:31:13.851711 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:13.851721 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:13.851772 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:13.851956 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:13.851957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:13.851959 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:13.852036 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:13.852038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.852690 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.852691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:13.853514 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:13.853516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.854132 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.854133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:13.854197 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:13.854199 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:31:13.854262 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 03:31:13.854304 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":44,"thread_start_us":146,"threads_started":1}
I20260927 03:31:13.854739 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1397066a6b98460791300c435af57363" candidate_uuid: "b77d4b143c8d4406881fc7c1654dbf61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" is_pre_election: true
I20260927 03:31:13.854931 16740 raft_consensus.cc:2454] T 1397066a6b98460791300c435af57363 P cb8b2d9f64ae44a0a6c65087194bf770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b77d4b143c8d4406881fc7c1654dbf61 in term 0.
I20260927 03:31:13.855048 16608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1397066a6b98460791300c435af57363" candidate_uuid: "b77d4b143c8d4406881fc7c1654dbf61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" is_pre_election: true
I20260927 03:31:13.855192 16401 leader_election.cc:302] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b77d4b143c8d4406881fc7c1654dbf61, cb8b2d9f64ae44a0a6c65087194bf770; no voters:
I20260927 03:31:13.855221 16608 raft_consensus.cc:2454] T 1397066a6b98460791300c435af57363 P 33fa80feb50e4c3eb17927e5ccc7142f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b77d4b143c8d4406881fc7c1654dbf61 in term 0.
I20260927 03:31:13.855317 16521 raft_consensus.cc:2781] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:31:13.855377 16521 raft_consensus.cc:482] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:31:13.855424 16521 raft_consensus.cc:3037] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:31:13.856175 16521 raft_consensus.cc:504] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33805 } } peers { permanent_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38421 } } peers { permanent_uuid: "b77d4b143c8d4406881fc7c1654dbf61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38477 } }
I20260927 03:31:13.856308 16521 leader_election.cc:288] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [CANDIDATE]: Term 1 election: Requested vote from peers cb8b2d9f64ae44a0a6c65087194bf770 (127.0.98.3:33805), 33fa80feb50e4c3eb17927e5ccc7142f (127.0.98.2:38421)
I20260927 03:31:13.856477 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1397066a6b98460791300c435af57363" candidate_uuid: "b77d4b143c8d4406881fc7c1654dbf61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb8b2d9f64ae44a0a6c65087194bf770"
I20260927 03:31:13.856526 16608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1397066a6b98460791300c435af57363" candidate_uuid: "b77d4b143c8d4406881fc7c1654dbf61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33fa80feb50e4c3eb17927e5ccc7142f"
I20260927 03:31:13.856594 16740 raft_consensus.cc:3037] T 1397066a6b98460791300c435af57363 P cb8b2d9f64ae44a0a6c65087194bf770 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:31:13.856613 16608 raft_consensus.cc:3037] T 1397066a6b98460791300c435af57363 P 33fa80feb50e4c3eb17927e5ccc7142f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:31:13.857267 16608 raft_consensus.cc:2454] T 1397066a6b98460791300c435af57363 P 33fa80feb50e4c3eb17927e5ccc7142f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b77d4b143c8d4406881fc7c1654dbf61 in term 1.
I20260927 03:31:13.857419 16740 raft_consensus.cc:2454] T 1397066a6b98460791300c435af57363 P cb8b2d9f64ae44a0a6c65087194bf770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b77d4b143c8d4406881fc7c1654dbf61 in term 1.
I20260927 03:31:13.857419 16401 leader_election.cc:302] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33fa80feb50e4c3eb17927e5ccc7142f, b77d4b143c8d4406881fc7c1654dbf61; no voters:
I20260927 03:31:13.857522 16521 raft_consensus.cc:2781] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:31:13.857692 16521 raft_consensus.cc:686] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [term 1 LEADER]: Becoming Leader. State: Replica: b77d4b143c8d4406881fc7c1654dbf61, State: Running, Role: LEADER
I20260927 03:31:13.857752 16521 consensus_queue.cc:237] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33805 } } peers { permanent_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38421 } } peers { permanent_uuid: "b77d4b143c8d4406881fc7c1654dbf61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38477 } }
I20260927 03:31:13.858579 16327 catalog_manager.cc:5495] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 reported cstate change: term changed from 0 to 1, leader changed from <none> to b77d4b143c8d4406881fc7c1654dbf61 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b77d4b143c8d4406881fc7c1654dbf61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b77d4b143c8d4406881fc7c1654dbf61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38477 } health_report { overall_health: HEALTHY } } }
I20260927 03:31:13.863756 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:13.861417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55650 (local address 127.0.98.1:38477)
0927 03:31:13.861472 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:13.861474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:13.861483 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:31:13.861555 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:13.861556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:13.861575 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:13.861660 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:31:13.861759 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.862220 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.862951 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:13.863163 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:13.863238 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:13.863401 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:31:13.863524 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:31:13.863601 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:13.863656 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 03:31:13.863678 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 03:31:13.865221 16740 raft_consensus.cc:1261] T 1397066a6b98460791300c435af57363 P cb8b2d9f64ae44a0a6c65087194bf770 [term 1 FOLLOWER]: Refusing update from remote peer b77d4b143c8d4406881fc7c1654dbf61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:31:13.865566 16608 raft_consensus.cc:1261] T 1397066a6b98460791300c435af57363 P 33fa80feb50e4c3eb17927e5ccc7142f [term 1 FOLLOWER]: Refusing update from remote peer b77d4b143c8d4406881fc7c1654dbf61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:31:13.865574 16521 consensus_queue.cc:1035] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:31:13.865850 16521 consensus_queue.cc:1035] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:31:13.869083 16804 mvcc.cc:205] Tried to move back new op lower bound from 7333805563349397504 to 7333805563321950208. Current Snapshot: MvccSnapshot[applied={T|T < 7333805563349397504 or (T in {7333805563349397504})}]
I20260927 03:31:13.870124 16327 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:31:13.870242 16327 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:31:13.870939 16327 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327: Sending DeleteTablet for 3 replicas of tablet 1397066a6b98460791300c435af57363
I20260927 03:31:13.871246 16449 tablet_service.cc:1514] Processing DeleteTablet for tablet 1397066a6b98460791300c435af57363 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:31:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55648
I20260927 03:31:13.871285 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 16365
I20260927 03:31:13.871343 16524 tablet_replica.cc:330] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61: stopping tablet replica
I20260927 03:31:13.871328 16720 tablet_service.cc:1514] Processing DeleteTablet for tablet 1397066a6b98460791300c435af57363 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:31:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45090
I20260927 03:31:13.871385 16524 raft_consensus.cc:2229] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:31:13.871491 16524 raft_consensus.cc:2258] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:31:13.871567 16588 tablet_service.cc:1514] Processing DeleteTablet for tablet 1397066a6b98460791300c435af57363 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:31:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33388
I20260927 03:31:13.871919 16814 tablet_replica.cc:330] T 1397066a6b98460791300c435af57363 P cb8b2d9f64ae44a0a6c65087194bf770: stopping tablet replica
I20260927 03:31:13.871981 16815 tablet_replica.cc:330] T 1397066a6b98460791300c435af57363 P 33fa80feb50e4c3eb17927e5ccc7142f: stopping tablet replica
I20260927 03:31:13.872014 16814 raft_consensus.cc:2229] T 1397066a6b98460791300c435af57363 P cb8b2d9f64ae44a0a6c65087194bf770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:31:13.872054 16815 raft_consensus.cc:2229] T 1397066a6b98460791300c435af57363 P 33fa80feb50e4c3eb17927e5ccc7142f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:31:13.872125 16814 raft_consensus.cc:2258] T 1397066a6b98460791300c435af57363 P cb8b2d9f64ae44a0a6c65087194bf770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:31:13.872133 16815 raft_consensus.cc:2258] T 1397066a6b98460791300c435af57363 P 33fa80feb50e4c3eb17927e5ccc7142f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:31:13.872504 16814 ts_tablet_manager.cc:1905] T 1397066a6b98460791300c435af57363 P cb8b2d9f64ae44a0a6c65087194bf770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:31:13.873198 16815 ts_tablet_manager.cc:1905] T 1397066a6b98460791300c435af57363 P 33fa80feb50e4c3eb17927e5ccc7142f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:31:13.874315 16814 ts_tablet_manager.cc:1918] T 1397066a6b98460791300c435af57363 P cb8b2d9f64ae44a0a6c65087194bf770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:31:13.874395 16814 log.cc:1196] T 1397066a6b98460791300c435af57363 P cb8b2d9f64ae44a0a6c65087194bf770: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/wal/wals/1397066a6b98460791300c435af57363
I20260927 03:31:13.874675 16814 ts_tablet_manager.cc:1939] T 1397066a6b98460791300c435af57363 P cb8b2d9f64ae44a0a6c65087194bf770: Deleting consensus metadata
I20260927 03:31:13.874994 16815 ts_tablet_manager.cc:1918] T 1397066a6b98460791300c435af57363 P 33fa80feb50e4c3eb17927e5ccc7142f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:31:13.875051 16815 log.cc:1196] T 1397066a6b98460791300c435af57363 P 33fa80feb50e4c3eb17927e5ccc7142f: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/wal/wals/1397066a6b98460791300c435af57363
I20260927 03:31:13.875129 16311 catalog_manager.cc:4841] TS cb8b2d9f64ae44a0a6c65087194bf770 (127.0.98.3:33805): tablet 1397066a6b98460791300c435af57363 (table test-table [id=590a61959fa74f08931e6e5d5888b2e6]) successfully deleted
I20260927 03:31:13.875276 16815 ts_tablet_manager.cc:1939] T 1397066a6b98460791300c435af57363 P 33fa80feb50e4c3eb17927e5ccc7142f: Deleting consensus metadata
I20260927 03:31:13.875921 16311 catalog_manager.cc:4841] TS 33fa80feb50e4c3eb17927e5ccc7142f (127.0.98.2:38421): tablet 1397066a6b98460791300c435af57363 (table test-table [id=590a61959fa74f08931e6e5d5888b2e6]) successfully deleted
W20260927 03:31:13.877576 16313 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38477 (error 108)
I20260927 03:31:13.877940 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 16520
I20260927 03:31:13.878084 16522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:13.877932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38477 (local address 127.0.0.1:55654)
0927 03:31:13.878002 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:13.878045 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 03:31:13.878755 16313 catalog_manager.cc:4584] TS b77d4b143c8d4406881fc7c1654dbf61 (127.0.98.1:38477): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1397066a6b98460791300c435af57363: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38477: connect: Connection refused (error 111)
I20260927 03:31:13.883033 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 16657
I20260927 03:31:13.887879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 16294
I20260927 03:31:33.893436 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35465
--webserver_interface=127.0.98.62
--webserver_port=36779
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:31:33.981024 16817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:33.981233 16817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:33.981256 16817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:33.982321 16817 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 03:31:33.982358 16817 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 03:31:33.982373 16817 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 03:31:33.982386 16817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:31:33.982401 16817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:33.982416 16817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:33.982429 16817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:31:33.982443 16817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:31:33.983914 16817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:33.984112 16817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:33.984289 16817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:31:33.989152 16823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:33.989298 16817 server_base.cc:1034] running on GCE node
W20260927 03:31:33.989181 16822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:33.989156 16825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:33.989678 16817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:33.990005 16817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:33.991212 16817 hybrid_clock.cc:648] HybridClock initialized: now 1790479893991194 us; error 36 us; skew 500 ppm
Sep 27 03:31:33 dist-test-slave-j0hf krb5kdc[15745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:31:34.000875 16817 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:31:34.001435 16817 webserver.cc:460] Webserver started at http://127.0.98.62:36779/ using document root <none> and password file <none>
I20260927 03:31:34.001701 16817 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:34.001775 16817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:34.003199 16817 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:31:34.005244 16832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:34.005505 16817 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:31:34.005638 16817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "ac032e00c98e4600a63fd0da0e315327"
format_stamp: "Formatted at 2026-09-27 03:30:52 on dist-test-slave-j0hf"
server_key: "5ee7528513c5b10ecf32bf3c3eea2548"
server_key_iv: "8abe229cc80596e3e7a28fdbc1e9bb3c"
server_key_version: "encryptionkey@0"
I20260927 03:31:34.006009 16817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:34.016790 16817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:34.018868 16817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:34.019016 16817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:34.023546 16817 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35465
I20260927 03:31:34.023579 16884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35465 every 8 connection(s)
I20260927 03:31:34.024019 16817 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:31:34.025550 16885 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:31:34.026262 16885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327: Bootstrap starting.
I20260927 03:31:34.029002 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 16817
I20260927 03:31:34.029341 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38477
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35465
--webserver_port=44781
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:31:34.036049 16885 log.cc:824] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327: Log is configured to *not* fsync() on all Append() calls
I20260927 03:31:34.038483 16885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:31:34.038806 16885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327: Bootstrap complete.
I20260927 03:31:34.041342 16885 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac032e00c98e4600a63fd0da0e315327" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35465 } }
I20260927 03:31:34.041600 16885 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac032e00c98e4600a63fd0da0e315327, State: Initialized, Role: FOLLOWER
I20260927 03:31:34.041797 16885 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac032e00c98e4600a63fd0da0e315327" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35465 } }
I20260927 03:31:34.041883 16885 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:31:34.041944 16885 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:31:34.042001 16885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [term 2 FOLLOWER]: Advancing to term 3
I20260927 03:31:34.043025 16885 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac032e00c98e4600a63fd0da0e315327" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35465 } }
I20260927 03:31:34.043159 16885 leader_election.cc:302] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac032e00c98e4600a63fd0da0e315327; no voters:
I20260927 03:31:34.043362 16885 leader_election.cc:288] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 03:31:34.043397 16889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 03:31:34.043537 16889 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [term 3 LEADER]: Becoming Leader. State: Replica: ac032e00c98e4600a63fd0da0e315327, State: Running, Role: LEADER
I20260927 03:31:34.043632 16889 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac032e00c98e4600a63fd0da0e315327" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35465 } }
I20260927 03:31:34.043673 16885 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:31:34.043952 16889 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac032e00c98e4600a63fd0da0e315327. Latest consensus state: current_term: 3 leader_uuid: "ac032e00c98e4600a63fd0da0e315327" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac032e00c98e4600a63fd0da0e315327" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35465 } } }
I20260927 03:31:34.044029 16889 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [sys.catalog]: This master's current role is: LEADER
I20260927 03:31:34.043955 16890 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ac032e00c98e4600a63fd0da0e315327" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac032e00c98e4600a63fd0da0e315327" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35465 } } }
I20260927 03:31:34.044068 16890 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327 [sys.catalog]: This master's current role is: LEADER
I20260927 03:31:34.044230 16893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:31:34.045183 16893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:31:34.045301 16893 catalog_manager.cc:1259] Loaded cluster ID: d2c2c459365d4fe2aeda94b99a865a1f
I20260927 03:31:34.045562 16893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:31:34.046854 16893 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:31:34.047041 16893 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327: Loaded TSK: 0
I20260927 03:31:34.047094 16893 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ac032e00c98e4600a63fd0da0e315327: Loaded TSK: 1
F20260927 03:31:34.047341 16893 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7facf99b69ad google::LogMessage::Fail() at ??:0
@ 0x7facf99b988a google::LogMessage::SendToLog() at ??:0
@ 0x7facf99b64de google::LogMessage::Flush() at ??:0
@ 0x7facf99b7a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7facfc123104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7facfc120d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fad019101cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fad0191048e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fad01910594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fad018d6c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fad018e0ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7facfb8840d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7facfb879954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7facfc6dd6db start_thread at ??:0
@ 0x7facf84b461f clone at ??:0
W20260927 03:31:34.124405 16887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:34.124642 16887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:34.124683 16887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:34.125896 16887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:34.125967 16887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:34.126015 16887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:31:34.127614 16887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38477
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44781
--tserver_master_addrs=127.0.98.62:35465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:34.127880 16887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:34.128124 16887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:31:34.134704 16927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:34.134739 16926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:34.134706 16929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:34.135181 16887 server_base.cc:1034] running on GCE node
I20260927 03:31:34.135403 16887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:34.135700 16887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:34.136879 16887 hybrid_clock.cc:648] HybridClock initialized: now 1790479894136866 us; error 34 us; skew 500 ppm
Sep 27 03:31:34 dist-test-slave-j0hf krb5kdc[15745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:31:34.139463 16887 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:31:34.139927 16887 webserver.cc:460] Webserver started at http://127.0.98.1:44781/ using document root <none> and password file <none>
I20260927 03:31:34.140169 16887 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:34.140242 16887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:34.141739 16887 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:31:34.142508 16936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:34.142695 16887 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:31:34.142776 16887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "b77d4b143c8d4406881fc7c1654dbf61"
format_stamp: "Formatted at 2026-09-27 03:30:52 on dist-test-slave-j0hf"
server_key: "c90e9ad6f2a4c8b27674a814971a62df"
server_key_iv: "acc6c050ad6c1c9f35c598926bb92998"
server_key_version: "encryptionkey@0"
I20260927 03:31:34.143157 16887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:34.156672 16887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:34.159085 16887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:34.159261 16887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:34.159614 16887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:31:34.160108 16943 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 03:31:34.161607 16887 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 03:31:34.161664 16887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 03:31:34.161691 16887 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 03:31:34.162382 16887 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 03:31:34.162425 16887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 03:31:34.162478 16943 tablet_bootstrap.cc:492] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61: Bootstrap starting.
I20260927 03:31:34.170372 16887 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38477
I20260927 03:31:34.170635 17050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38477 every 8 connection(s)
I20260927 03:31:34.170943 16887 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:31:34.174567 16943 log.cc:824] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61: Log is configured to *not* fsync() on all Append() calls
I20260927 03:31:34.175026 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 16887
I20260927 03:31:34.175338 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38421
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35465
--webserver_port=33211
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:31:34.178157 16943 tablet_bootstrap.cc:492] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:31:34.178505 16943 tablet_bootstrap.cc:492] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61: Bootstrap complete.
I20260927 03:31:34.179792 16943 ts_tablet_manager.cc:1397] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.008s
I20260927 03:31:34.181591 16943 raft_consensus.cc:348] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33805 } } peers { permanent_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38421 } } peers { permanent_uuid: "b77d4b143c8d4406881fc7c1654dbf61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38477 } }
I20260927 03:31:34.181981 16943 raft_consensus.cc:729] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b77d4b143c8d4406881fc7c1654dbf61, State: Initialized, Role: FOLLOWER
I20260927 03:31:34.182155 16943 consensus_queue.cc:260] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33805 } } peers { permanent_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38421 } } peers { permanent_uuid: "b77d4b143c8d4406881fc7c1654dbf61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38477 } }
I20260927 03:31:34.182466 16943 ts_tablet_manager.cc:1428] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
W20260927 03:31:34.259529 17055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:34.259755 17055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:34.259794 17055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:34.260809 17053 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:31:34.171713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.1:49571)
0927 03:31:34.171902 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:34.171948 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:34.172772 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:34.260599 (+ 87827us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":112,"threads_started":1}
W20260927 03:31:34.261116 17051 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35465 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35465: Client connection negotiation failed: client connection to 127.0.98.62:35465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 03:31:34.261231 17055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:34.261282 17055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:34.261337 17055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:31:34.261539 17053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:34.261420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.1:52733)
0927 03:31:34.261466 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:34.261516 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 03:31:34.261866 17053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:34.261770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.1:50235)
0927 03:31:34.261818 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:34.261846 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 03:31:34.261984 17051 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 03:31:34.263036 17055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33211
--tserver_master_addrs=127.0.98.62:35465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:34.263291 17055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:34.263556 17055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:31:34.268451 17062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:34.268424 17061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:34.268474 17064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:34.268642 17055 server_base.cc:1034] running on GCE node
I20260927 03:31:34.268847 17055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:34.269057 17055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:34.270195 17055 hybrid_clock.cc:648] HybridClock initialized: now 1790479894270182 us; error 31 us; skew 500 ppm
Sep 27 03:31:34 dist-test-slave-j0hf krb5kdc[15745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:31:34.272634 17055 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:31:34.273073 17055 webserver.cc:460] Webserver started at http://127.0.98.2:33211/ using document root <none> and password file <none>
I20260927 03:31:34.273310 17055 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:34.273379 17055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:34.274721 17055 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:31:34.275342 17071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:34.275622 17055 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:31:34.275717 17055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "33fa80feb50e4c3eb17927e5ccc7142f"
format_stamp: "Formatted at 2026-09-27 03:30:52 on dist-test-slave-j0hf"
server_key: "1a539d321fc934b389ee8dd631a6767e"
server_key_iv: "f88e40753c245127a2457df7b6f29ef6"
server_key_version: "encryptionkey@0"
I20260927 03:31:34.276060 17055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:34.309087 17055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:34.311025 17055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:34.311205 17055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:34.311496 17055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:31:34.311825 17055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:31:34.311878 17055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:34.311920 17055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:31:34.311949 17055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:34.317806 17055 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38421
I20260927 03:31:34.317848 17184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38421 every 8 connection(s)
I20260927 03:31:34.318178 17055 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:31:34.319211 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:34.318842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.2:54781)
0927 03:31:34.318991 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:34.319056 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
W20260927 03:31:34.319638 17185 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35465 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35465: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
I20260927 03:31:34.319904 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:34.319811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.2:56899)
0927 03:31:34.319858 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:34.319882 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 03:31:34.320185 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:34.320089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.2:50819)
0927 03:31:34.320144 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:34.320166 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 03:31:34.320292 17185 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 03:31:34.320775 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 17055
I20260927 03:31:34.321059 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33805
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35465
--webserver_port=42765
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:31:34.403620 17188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:34.403847 17188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:34.403877 17188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:34.405083 17188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:34.405131 17188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:34.405161 17188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:31:34.406643 17188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33805
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42765
--tserver_master_addrs=127.0.98.62:35465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:34.406917 17188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:34.407159 17188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:31:34.412340 17194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:34.412346 17193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:34.412356 17196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:34.412674 17188 server_base.cc:1034] running on GCE node
I20260927 03:31:34.413007 17188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:34.413280 17188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:34.414466 17188 hybrid_clock.cc:648] HybridClock initialized: now 1790479894414448 us; error 45 us; skew 500 ppm
Sep 27 03:31:34 dist-test-slave-j0hf krb5kdc[15745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:31:34.417059 17188 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:31:34.417572 17188 webserver.cc:460] Webserver started at http://127.0.98.3:42765/ using document root <none> and password file <none>
I20260927 03:31:34.417824 17188 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:34.417896 17188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:34.419263 17188 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:31:34.419947 17203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:34.420115 17188 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:34.420217 17188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "cb8b2d9f64ae44a0a6c65087194bf770"
format_stamp: "Formatted at 2026-09-27 03:30:52 on dist-test-slave-j0hf"
server_key: "96524ede5ba89976c92496a281b35074"
server_key_iv: "3998a782d2aba15a960a0a4bd54ffede"
server_key_version: "encryptionkey@0"
I20260927 03:31:34.420583 17188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:34.441651 17188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:34.443789 17188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:34.443956 17188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:34.444203 17188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:31:34.444533 17188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:31:34.444585 17188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:34.444618 17188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:31:34.444657 17188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:34.450743 17188 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33805
I20260927 03:31:34.450798 17316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33805 every 8 connection(s)
I20260927 03:31:34.451176 17188 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:31:34.452332 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:34.451964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.3:42533)
0927 03:31:34.452129 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:34.452196 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":77,"threads_started":1}
W20260927 03:31:34.452710 17317 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35465 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35465: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
I20260927 03:31:34.452989 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:34.452897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.3:43665)
0927 03:31:34.452950 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:34.452974 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 03:31:34.453227 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:34.453179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.3:43313)
0927 03:31:34.453198 (+ 19us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:34.453215 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260927 03:31:34.453356 17317 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 03:31:34.456099 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 17188
I20260927 03:31:35.262599 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:35.262336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.1:34141)
0927 03:31:35.262484 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:35.262522 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":62,"threads_started":1}
I20260927 03:31:35.320837 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:35.320618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.2:33357)
0927 03:31:35.320745 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:35.320785 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":55,"threads_started":1}
I20260927 03:31:35.453893 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:35.453691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.3:41655)
0927 03:31:35.453806 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:35.453843 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":56,"threads_started":1}
I20260927 03:31:35.873585 17324 raft_consensus.cc:482] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:31:35.873714 17324 raft_consensus.cc:504] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33805 } } peers { permanent_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38421 } } peers { permanent_uuid: "b77d4b143c8d4406881fc7c1654dbf61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38477 } }
I20260927 03:31:35.874135 17324 leader_election.cc:288] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cb8b2d9f64ae44a0a6c65087194bf770 (127.0.98.3:33805), 33fa80feb50e4c3eb17927e5ccc7142f (127.0.98.2:38421)
Sep 27 03:31:35 dist-test-slave-j0hf krb5kdc[15745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 27 03:31:35 dist-test-slave-j0hf krb5kdc[15745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260927 03:31:35.885051 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:35.874378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33805 (local address 127.0.98.1:35749)
0927 03:31:35.874495 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:35.874507 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:35.874566 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:35.875490 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:35.875493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:35.875826 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:31:35.876033 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:35.876040 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:35.877108 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:35.877109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:35.877862 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:35.877864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:35.878493 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:35.878494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:35.878603 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:35.879268 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:31:35.879273 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:31:35.881291 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:31:35.883400 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:31:35.883401 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:31:35.883402 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:31:35.883695 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:31:35.883973 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:31:35.883974 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:31:35.883974 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:31:35.884100 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:31:35.884506 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:31:35.884513 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:31:35.884648 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 03:31:35.884750 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 03:31:35.884952 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":45,"thread_start_us":50,"threads_started":1}
I20260927 03:31:35.885063 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:35.874494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35749 (local address 127.0.98.3:33805)
0927 03:31:35.874695 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:35.874708 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:35.874750 (+ 42us) server_negotiation.cc:407] Connection header received
0927 03:31:35.875220 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:35.875222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:35.875264 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:35.875434 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:35.876150 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:35.876999 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:35.877964 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:35.878385 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:35.878521 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:35.881399 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:35.881402 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:35.881403 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:31:35.881414 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:35.883305 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:35.883787 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:35.883790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:35.883790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:35.883901 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:35.884174 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:35.884175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:35.884175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:35.884429 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:35.884526 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:35.884816 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 03:31:35.884940 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":122,"threads_started":1}
I20260927 03:31:35.885113 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:35.874712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40673 (local address 127.0.98.2:38421)
0927 03:31:35.874831 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:35.874839 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:35.874858 (+ 19us) server_negotiation.cc:407] Connection header received
0927 03:31:35.875349 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:35.875351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:35.875392 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:35.875602 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:35.876194 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:35.877055 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:35.877978 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:35.878428 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:35.878578 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:35.881063 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:35.881067 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:35.881067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:31:35.881106 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:35.883157 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:35.883768 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:35.883771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:35.883772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:35.883869 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:35.884178 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:35.884179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:35.884179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:35.884429 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:35.884534 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:35.884823 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 03:31:35.884977 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":48,"threads_started":1}
I20260927 03:31:35.885183 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:35.874538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38421 (local address 127.0.98.1:40673)
0927 03:31:35.874702 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:35.874711 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:35.874753 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:35.875585 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:35.875586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:35.875840 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:31:35.876034 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:35.876040 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:35.877183 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:35.877184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:35.877861 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:35.877863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:35.878545 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:35.878547 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:35.878622 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:35.879224 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:31:35.879233 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:31:35.880922 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:31:35.883263 (+ 2341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:31:35.883271 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:31:35.883291 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:31:35.883667 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:31:35.883967 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:31:35.883968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:31:35.883969 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:31:35.884101 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:31:35.884515 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:31:35.884516 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:31:35.884649 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 03:31:35.884761 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 03:31:35.885039 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":151,"thread_start_us":106,"threads_started":1}
W20260927 03:31:35.886444 16937 leader_election.cc:341] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb8b2d9f64ae44a0a6c65087194bf770 (127.0.98.3:33805): Not found: Tablet not found: 1397066a6b98460791300c435af57363
W20260927 03:31:35.886694 16937 leader_election.cc:341] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 33fa80feb50e4c3eb17927e5ccc7142f (127.0.98.2:38421): Not found: Tablet not found: 1397066a6b98460791300c435af57363
I20260927 03:31:35.886749 16937 leader_election.cc:302] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b77d4b143c8d4406881fc7c1654dbf61; no voters: 33fa80feb50e4c3eb17927e5ccc7142f, cb8b2d9f64ae44a0a6c65087194bf770
I20260927 03:31:35.886883 17324 raft_consensus.cc:2726] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:31:36.263266 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:36.263115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.1:56027)
0927 03:31:36.263197 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:36.263228 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 03:31:36.321561 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:36.321328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.2:52035)
0927 03:31:36.321463 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:36.321516 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":58,"threads_started":1}
I20260927 03:31:36.454547 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:36.454313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.3:53973)
0927 03:31:36.454469 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:36.454507 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":94,"threads_started":1}
I20260927 03:31:37.264149 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:37.263902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.1:37217)
0927 03:31:37.264051 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:37.264093 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260927 03:31:37.322324 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:37.322074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.2:45925)
0927 03:31:37.322242 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:37.322284 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
I20260927 03:31:37.455245 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:37.455004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.3:58955)
0927 03:31:37.455158 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:37.455198 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":69,"threads_started":1}
I20260927 03:31:37.810115 17334 raft_consensus.cc:482] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:31:37.810230 17334 raft_consensus.cc:504] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33805 } } peers { permanent_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38421 } } peers { permanent_uuid: "b77d4b143c8d4406881fc7c1654dbf61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38477 } }
I20260927 03:31:37.810448 17334 leader_election.cc:288] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cb8b2d9f64ae44a0a6c65087194bf770 (127.0.98.3:33805), 33fa80feb50e4c3eb17927e5ccc7142f (127.0.98.2:38421)
W20260927 03:31:37.810979 16937 leader_election.cc:341] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb8b2d9f64ae44a0a6c65087194bf770 (127.0.98.3:33805): Not found: Tablet not found: 1397066a6b98460791300c435af57363
W20260927 03:31:37.811048 16937 leader_election.cc:341] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 33fa80feb50e4c3eb17927e5ccc7142f (127.0.98.2:38421): Not found: Tablet not found: 1397066a6b98460791300c435af57363
I20260927 03:31:37.811101 16937 leader_election.cc:302] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b77d4b143c8d4406881fc7c1654dbf61; no voters: 33fa80feb50e4c3eb17927e5ccc7142f, cb8b2d9f64ae44a0a6c65087194bf770
I20260927 03:31:37.811201 17334 raft_consensus.cc:2726] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:31:38.265012 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:38.264711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.1:60575)
0927 03:31:38.264883 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:38.264931 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":80,"threads_started":1}
I20260927 03:31:38.323046 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:38.322815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.2:53157)
0927 03:31:38.322948 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:38.322985 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260927 03:31:38.455979 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:38.455740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.3:45021)
0927 03:31:38.455890 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:38.455932 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
I20260927 03:31:39.265910 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:39.265592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.1:56589)
0927 03:31:39.265827 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:39.265863 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":77,"threads_started":1}
I20260927 03:31:39.323786 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:39.323586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.2:49805)
0927 03:31:39.323702 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:39.323746 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
I20260927 03:31:39.456808 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:39.456579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.3:46023)
0927 03:31:39.456725 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:39.456771 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":62,"threads_started":1}
I20260927 03:31:39.511601 17341 raft_consensus.cc:482] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:31:39.511698 17341 raft_consensus.cc:504] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33805 } } peers { permanent_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38421 } } peers { permanent_uuid: "b77d4b143c8d4406881fc7c1654dbf61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38477 } }
I20260927 03:31:39.511929 17341 leader_election.cc:288] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cb8b2d9f64ae44a0a6c65087194bf770 (127.0.98.3:33805), 33fa80feb50e4c3eb17927e5ccc7142f (127.0.98.2:38421)
W20260927 03:31:39.512406 16937 leader_election.cc:341] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb8b2d9f64ae44a0a6c65087194bf770 (127.0.98.3:33805): Not found: Tablet not found: 1397066a6b98460791300c435af57363
W20260927 03:31:39.512485 16937 leader_election.cc:341] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 33fa80feb50e4c3eb17927e5ccc7142f (127.0.98.2:38421): Not found: Tablet not found: 1397066a6b98460791300c435af57363
I20260927 03:31:39.512501 16937 leader_election.cc:302] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b77d4b143c8d4406881fc7c1654dbf61; no voters: 33fa80feb50e4c3eb17927e5ccc7142f, cb8b2d9f64ae44a0a6c65087194bf770
I20260927 03:31:39.512620 17341 raft_consensus.cc:2726] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:31:40.266757 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:40.266493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.1:53413)
0927 03:31:40.266657 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:40.266699 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":87,"threads_started":1}
I20260927 03:31:40.324424 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:40.324217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.2:50155)
0927 03:31:40.324350 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:40.324386 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20260927 03:31:40.457468 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:40.457235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.3:46731)
0927 03:31:40.457389 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:40.457424 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":84,"threads_started":1}
I20260927 03:31:41.267587 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:41.267277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.1:60377)
0927 03:31:41.267414 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:41.267537 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":65,"threads_started":1}
I20260927 03:31:41.325089 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:41.324877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.2:58755)
0927 03:31:41.325004 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:41.325044 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
I20260927 03:31:41.458153 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:41.457935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.3:40627)
0927 03:31:41.458073 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:41.458106 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260927 03:31:42.268358 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:42.268126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.1:55441)
0927 03:31:42.268253 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:42.268295 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260927 03:31:42.325788 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:42.325560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.2:56313)
0927 03:31:42.325704 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:42.325744 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":75,"threads_started":1}
I20260927 03:31:42.458850 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:42.458618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.3:41023)
0927 03:31:42.458743 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:42.458785 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":57,"threads_started":1}
I20260927 03:31:42.767592 17351 raft_consensus.cc:482] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:31:42.767716 17351 raft_consensus.cc:504] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33805 } } peers { permanent_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38421 } } peers { permanent_uuid: "b77d4b143c8d4406881fc7c1654dbf61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38477 } }
I20260927 03:31:42.767920 17351 leader_election.cc:288] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cb8b2d9f64ae44a0a6c65087194bf770 (127.0.98.3:33805), 33fa80feb50e4c3eb17927e5ccc7142f (127.0.98.2:38421)
W20260927 03:31:42.768426 16937 leader_election.cc:341] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb8b2d9f64ae44a0a6c65087194bf770 (127.0.98.3:33805): Not found: Tablet not found: 1397066a6b98460791300c435af57363
W20260927 03:31:42.768513 16937 leader_election.cc:341] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 33fa80feb50e4c3eb17927e5ccc7142f (127.0.98.2:38421): Not found: Tablet not found: 1397066a6b98460791300c435af57363
I20260927 03:31:42.768533 16937 leader_election.cc:302] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b77d4b143c8d4406881fc7c1654dbf61; no voters: 33fa80feb50e4c3eb17927e5ccc7142f, cb8b2d9f64ae44a0a6c65087194bf770
I20260927 03:31:42.768648 17351 raft_consensus.cc:2726] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:31:43.269182 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:43.268944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.1:53475)
0927 03:31:43.269074 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:43.269117 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":60,"threads_started":1}
I20260927 03:31:43.326556 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:43.326321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.2:59369)
0927 03:31:43.326463 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:43.326510 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":67,"threads_started":1}
I20260927 03:31:43.459530 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:43.459298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.3:43545)
0927 03:31:43.459445 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:43.459485 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":62,"threads_started":1}
I20260927 03:31:44.269936 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:44.269704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.1:55807)
0927 03:31:44.269843 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:44.269879 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260927 03:31:44.327399 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:44.327121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.2:58239)
0927 03:31:44.327281 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:44.327344 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":65,"threads_started":1}
I20260927 03:31:44.460247 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:44.459992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.3:37943)
0927 03:31:44.460156 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:44.460199 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":85,"threads_started":1}
I20260927 03:31:45.270637 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:45.270404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.1:44163)
0927 03:31:45.270554 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:45.270588 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":75,"threads_started":1}
I20260927 03:31:45.328289 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:45.328047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.2:52841)
0927 03:31:45.328192 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:45.328234 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":73,"threads_started":1}
I20260927 03:31:45.460961 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:45.460745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.3:56281)
0927 03:31:45.460876 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:45.460913 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
I20260927 03:31:46.271301 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:46.271097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.1:56841)
0927 03:31:46.271220 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:46.271254 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260927 03:31:46.329129 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:46.328900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.2:48845)
0927 03:31:46.329038 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:46.329083 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":57,"threads_started":1}
I20260927 03:31:46.461760 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:46.461540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.3:59207)
0927 03:31:46.461684 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:46.461716 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
I20260927 03:31:47.272011 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:47.271794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.1:45379)
0927 03:31:47.271928 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:47.271963 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":74,"threads_started":1}
I20260927 03:31:47.310880 17365 raft_consensus.cc:482] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:31:47.310977 17365 raft_consensus.cc:504] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8b2d9f64ae44a0a6c65087194bf770" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33805 } } peers { permanent_uuid: "33fa80feb50e4c3eb17927e5ccc7142f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38421 } } peers { permanent_uuid: "b77d4b143c8d4406881fc7c1654dbf61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38477 } }
I20260927 03:31:47.311182 17365 leader_election.cc:288] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cb8b2d9f64ae44a0a6c65087194bf770 (127.0.98.3:33805), 33fa80feb50e4c3eb17927e5ccc7142f (127.0.98.2:38421)
W20260927 03:31:47.311646 16937 leader_election.cc:341] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 33fa80feb50e4c3eb17927e5ccc7142f (127.0.98.2:38421): Not found: Tablet not found: 1397066a6b98460791300c435af57363
W20260927 03:31:47.311719 16937 leader_election.cc:341] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb8b2d9f64ae44a0a6c65087194bf770 (127.0.98.3:33805): Not found: Tablet not found: 1397066a6b98460791300c435af57363
I20260927 03:31:47.311738 16937 leader_election.cc:302] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b77d4b143c8d4406881fc7c1654dbf61; no voters: 33fa80feb50e4c3eb17927e5ccc7142f, cb8b2d9f64ae44a0a6c65087194bf770
I20260927 03:31:47.311815 17365 raft_consensus.cc:2726] T 1397066a6b98460791300c435af57363 P b77d4b143c8d4406881fc7c1654dbf61 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 03:31:47.329831 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:47.329613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.2:36913)
0927 03:31:47.329749 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:47.329789 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
I20260927 03:31:47.462481 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:47.462282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.3:51629)
0927 03:31:47.462406 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:47.462438 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20260927 03:31:48.272662 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:48.272434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.1:46313)
0927 03:31:48.272577 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:48.272614 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":62,"threads_started":1}
I20260927 03:31:48.330577 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:48.330354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.2:60311)
0927 03:31:48.330485 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:48.330529 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260927 03:31:48.463310 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:48.462977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.3:40989)
0927 03:31:48.463141 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:48.463226 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":85,"threads_started":1}
I20260927 03:31:49.273445 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:49.273192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.1:53889)
0927 03:31:49.273350 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:49.273383 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":109,"threads_started":1}
I20260927 03:31:49.331393 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:49.331195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.2:48481)
0927 03:31:49.331305 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:49.331340 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":67,"threads_started":1}
I20260927 03:31:49.460206 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 16887
I20260927 03:31:49.464036 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:49.463851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35465 (local address 127.0.98.3:58869)
0927 03:31:49.463966 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:49.463998 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":64,"threads_started":1}
I20260927 03:31:49.465370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 17055
I20260927 03:31:49.469743 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 17188
2026-09-27T03:31:49Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58461 ms)
[----------] 30 tests from SecurityITest (233611 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:31:49 dist-test-slave-j0hf krb5kdc[17376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:31:49 dist-test-slave-j0hf krb5kdc[17376](info): set up 2 sockets
Sep 27 03:31:49 dist-test-slave-j0hf krb5kdc[17376](info): commencing operation
krb5kdc: starting...
W20260927 03:31:50.519060 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:31:50 dist-test-slave-j0hf krb5kdc[17376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:31:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:31:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:31:50.634261 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40219
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:31:50.715056 17389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:50.715267 17389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:50.715289 17389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:50.716521 17389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:31:50.716564 17389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:50.716579 17389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:50.716627 17389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:31:50.716655 17389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:31:50.718103 17389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:50.718358 17389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:50.718624 17389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:31:50.723623 17389 server_base.cc:1034] running on GCE node
W20260927 03:31:50.723654 17394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:50.723654 17397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:50.723738 17395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:50.724114 17389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:50.724359 17389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:50.725494 17389 hybrid_clock.cc:648] HybridClock initialized: now 1790479910725476 us; error 38 us; skew 500 ppm
Sep 27 03:31:50 dist-test-slave-j0hf krb5kdc[17376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:31:50.727991 17389 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:31:50.728454 17389 webserver.cc:460] Webserver started at http://127.0.98.62:37377/ using document root <none> and password file <none>
I20260927 03:31:50.728691 17389 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:50.728778 17389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:50.728937 17389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:31:50.729988 17389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "c3155d3c1d074ff48a10bbf72beb1a44"
format_stamp: "Formatted at 2026-09-27 03:31:50 on dist-test-slave-j0hf"
server_key: "505a1eac885024ef0bb101cd79eb6f80"
server_key_iv: "70b17ca410a3400b0be79a3c610c00d2"
server_key_version: "encryptionkey@0"
I20260927 03:31:50.730355 17389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "c3155d3c1d074ff48a10bbf72beb1a44"
format_stamp: "Formatted at 2026-09-27 03:31:50 on dist-test-slave-j0hf"
server_key: "505a1eac885024ef0bb101cd79eb6f80"
server_key_iv: "70b17ca410a3400b0be79a3c610c00d2"
server_key_version: "encryptionkey@0"
I20260927 03:31:50.731735 17389 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:31:50.732450 17404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:50.732609 17389 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:31:50.732697 17389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "c3155d3c1d074ff48a10bbf72beb1a44"
format_stamp: "Formatted at 2026-09-27 03:31:50 on dist-test-slave-j0hf"
server_key: "505a1eac885024ef0bb101cd79eb6f80"
server_key_iv: "70b17ca410a3400b0be79a3c610c00d2"
server_key_version: "encryptionkey@0"
I20260927 03:31:50.732779 17389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:50.745643 17389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:50.747610 17389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:50.747777 17389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:50.751747 17389 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40219
I20260927 03:31:50.751781 17456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40219 every 8 connection(s)
I20260927 03:31:50.752144 17389 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:31:50.753008 17457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:31:50.755203 17457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3155d3c1d074ff48a10bbf72beb1a44: Bootstrap starting.
I20260927 03:31:50.755815 17457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3155d3c1d074ff48a10bbf72beb1a44: Neither blocks nor log segments found. Creating new log.
I20260927 03:31:50.756049 17457 log.cc:824] T 00000000000000000000000000000000 P c3155d3c1d074ff48a10bbf72beb1a44: Log is configured to *not* fsync() on all Append() calls
I20260927 03:31:50.756671 17457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3155d3c1d074ff48a10bbf72beb1a44: No bootstrap required, opened a new log
I20260927 03:31:50.758213 17457 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3155d3c1d074ff48a10bbf72beb1a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3155d3c1d074ff48a10bbf72beb1a44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40219 } }
I20260927 03:31:50.758352 17457 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3155d3c1d074ff48a10bbf72beb1a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:31:50.758375 17457 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3155d3c1d074ff48a10bbf72beb1a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3155d3c1d074ff48a10bbf72beb1a44, State: Initialized, Role: FOLLOWER
I20260927 03:31:50.758490 17457 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3155d3c1d074ff48a10bbf72beb1a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3155d3c1d074ff48a10bbf72beb1a44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40219 } }
I20260927 03:31:50.758543 17457 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3155d3c1d074ff48a10bbf72beb1a44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:31:50.758563 17457 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3155d3c1d074ff48a10bbf72beb1a44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:31:50.758590 17457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3155d3c1d074ff48a10bbf72beb1a44 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:31:50.759059 17457 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3155d3c1d074ff48a10bbf72beb1a44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3155d3c1d074ff48a10bbf72beb1a44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40219 } }
I20260927 03:31:50.759202 17457 leader_election.cc:302] T 00000000000000000000000000000000 P c3155d3c1d074ff48a10bbf72beb1a44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3155d3c1d074ff48a10bbf72beb1a44; no voters:
I20260927 03:31:50.759291 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 17389
I20260927 03:31:50.759385 17457 leader_election.cc:288] T 00000000000000000000000000000000 P c3155d3c1d074ff48a10bbf72beb1a44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:31:50.759411 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:31:50.759413 17460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3155d3c1d074ff48a10bbf72beb1a44 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:31:50.759614 17460 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3155d3c1d074ff48a10bbf72beb1a44 [term 1 LEADER]: Becoming Leader. State: Replica: c3155d3c1d074ff48a10bbf72beb1a44, State: Running, Role: LEADER
I20260927 03:31:50.759647 392 external_mini_cluster.cc:1428] Setting key 7a703486a27a0ec5219b2be753c145aa
I20260927 03:31:50.759685 17457 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3155d3c1d074ff48a10bbf72beb1a44 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:31:50.759734 17460 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3155d3c1d074ff48a10bbf72beb1a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3155d3c1d074ff48a10bbf72beb1a44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40219 } }
I20260927 03:31:50.760109 17460 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3155d3c1d074ff48a10bbf72beb1a44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3155d3c1d074ff48a10bbf72beb1a44. Latest consensus state: current_term: 1 leader_uuid: "c3155d3c1d074ff48a10bbf72beb1a44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3155d3c1d074ff48a10bbf72beb1a44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40219 } } }
I20260927 03:31:50.760259 17461 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3155d3c1d074ff48a10bbf72beb1a44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3155d3c1d074ff48a10bbf72beb1a44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3155d3c1d074ff48a10bbf72beb1a44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40219 } } }
I20260927 03:31:50.760267 17460 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3155d3c1d074ff48a10bbf72beb1a44 [sys.catalog]: This master's current role is: LEADER
I20260927 03:31:50.760342 17461 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3155d3c1d074ff48a10bbf72beb1a44 [sys.catalog]: This master's current role is: LEADER
I20260927 03:31:50.760705 17471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:31:50.761755 17471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:31:50.763125 17471 catalog_manager.cc:1347] Generated new cluster ID: 24778f02ae674bb393a523a723568f0a
I20260927 03:31:50.763175 17471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:31:50 dist-test-slave-j0hf krb5kdc[17376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:31:50.771996 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:50.760308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44800 (local address 127.0.98.62:40219)
0927 03:31:50.760535 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:50.760544 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:50.760604 (+ 60us) server_negotiation.cc:407] Connection header received
0927 03:31:50.760903 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:50.760926 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:50.760994 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:50.761187 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:50.761960 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:50.763327 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:50.764427 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:50.765061 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:50.765177 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:50.767999 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:50.768004 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:50.768004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:31:50.768017 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:50.770203 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:50.770618 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:50.770622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:50.770622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:50.770754 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:50.770977 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:50.770978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:50.770978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:50.771323 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:50.771532 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:50.771617 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 03:31:50.771761 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":71,"threads_started":1}
I20260927 03:31:50.792649 17471 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:31:50.793210 17471 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:31:50.797484 17471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3155d3c1d074ff48a10bbf72beb1a44: Generated new TSK 0
I20260927 03:31:50.797688 17471 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:31:50.825881 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40219
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:31:50.907352 17483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:50.907635 17483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:50.907660 17483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:50.908826 17483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:50.908869 17483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:50.908895 17483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:31:50.910393 17483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:50.910684 17483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:50.910918 17483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:31:50.915781 17491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:50.915769 17489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:50.915766 17488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:50.916013 17483 server_base.cc:1034] running on GCE node
I20260927 03:31:50.916245 17483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:50.916432 17483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:50.917567 17483 hybrid_clock.cc:648] HybridClock initialized: now 1790479910917557 us; error 25 us; skew 500 ppm
Sep 27 03:31:50 dist-test-slave-j0hf krb5kdc[17376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:31:50.920039 17483 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:31:50.920554 17483 webserver.cc:460] Webserver started at http://127.0.98.1:38793/ using document root <none> and password file <none>
I20260927 03:31:50.920769 17483 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:50.920812 17483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:50.920974 17483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:31:50.921983 17483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-0/data/instance:
uuid: "1ba160786a3c45fbbba1cb74544c37d1"
format_stamp: "Formatted at 2026-09-27 03:31:50 on dist-test-slave-j0hf"
server_key: "b733ef1a72d363374b04eaa3fabca037"
server_key_iv: "737d31d56a8841bbfe0daaac8eaf8fbd"
server_key_version: "encryptionkey@0"
I20260927 03:31:50.922326 17483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1ba160786a3c45fbbba1cb74544c37d1"
format_stamp: "Formatted at 2026-09-27 03:31:50 on dist-test-slave-j0hf"
server_key: "b733ef1a72d363374b04eaa3fabca037"
server_key_iv: "737d31d56a8841bbfe0daaac8eaf8fbd"
server_key_version: "encryptionkey@0"
I20260927 03:31:50.923729 17483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:31:50.924425 17498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:50.924592 17483 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:31:50.924690 17483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "1ba160786a3c45fbbba1cb74544c37d1"
format_stamp: "Formatted at 2026-09-27 03:31:50 on dist-test-slave-j0hf"
server_key: "b733ef1a72d363374b04eaa3fabca037"
server_key_iv: "737d31d56a8841bbfe0daaac8eaf8fbd"
server_key_version: "encryptionkey@0"
I20260927 03:31:50.924767 17483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:50.940399 17483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:50.942242 17483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:50.942406 17483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:50.942638 17483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:31:50.942957 17483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:31:50.943009 17483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:50.943042 17483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:31:50.943068 17483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:50.948860 17483 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37787
I20260927 03:31:50.948904 17611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37787 every 8 connection(s)
I20260927 03:31:50.949267 17483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:31:50.951259 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 17483
I20260927 03:31:50.951396 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:31:50.951639 392 external_mini_cluster.cc:1428] Setting key 9d19c53058f9491d612ec089d0968a1d
Sep 27 03:31:50 dist-test-slave-j0hf krb5kdc[17376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:31:50.959784 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:50.950050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50281 (local address 127.0.98.62:40219)
0927 03:31:50.950111 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:50.950113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:50.950278 (+ 165us) server_negotiation.cc:407] Connection header received
0927 03:31:50.951222 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:50.951223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:50.951235 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:50.951319 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:50.952180 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:50.952949 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:50.954229 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:50.954422 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:50.954488 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:50.956875 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:50.956876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:50.956877 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:31:50.956885 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:50.958287 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:50.958692 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:50.958693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:50.958694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:50.958764 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:50.959016 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:50.959017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:50.959017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:50.959196 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:50.959274 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:50.959613 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 03:31:50.959698 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 03:31:50.959971 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:50.949997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40219 (local address 127.0.98.1:50281)
0927 03:31:50.950154 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:50.950195 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:50.951108 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:50.951414 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:50.951464 (+ 50us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:50.951815 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:31:50.952106 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:50.952112 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:50.953042 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:50.953043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:50.954138 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:50.954141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:50.954517 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:50.954518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:50.954667 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:50.955147 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:31:50.955157 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:31:50.956766 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:31:50.958371 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:31:50.958378 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:31:50.958380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:31:50.958601 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:31:50.958828 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:31:50.958829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:31:50.958829 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:31:50.958943 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:31:50.959278 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:31:50.959284 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:31:50.959435 (+ 151us) client_negotiation.cc:769] Sending connection context
0927 03:31:50.959577 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 03:31:50.959729 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260927 03:31:50.960515 17612 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40219
I20260927 03:31:50.960620 17612 heartbeater.cc:464] Registering TS with master...
I20260927 03:31:50.960806 17612 heartbeater.cc:511] Master 127.0.98.62:40219 requested a full tablet report, sending...
I20260927 03:31:50.961382 17421 ts_manager.cc:194] Registered new tserver with Master: 1ba160786a3c45fbbba1cb74544c37d1 (127.0.98.1:37787)
I20260927 03:31:50.962070 17421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50281
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:31:50.976164 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40219
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:31:51.055984 17617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:51.056191 17617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:51.056214 17617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:51.057400 17617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:51.057452 17617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:51.057479 17617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:31:51.059013 17617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:51.059252 17617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:51.059540 17617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:31:51.064672 17622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:51.064630 17625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:51.064754 17617 server_base.cc:1034] running on GCE node
W20260927 03:31:51.064630 17623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:51.065127 17617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:51.065330 17617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:51.066493 17617 hybrid_clock.cc:648] HybridClock initialized: now 1790479911066474 us; error 48 us; skew 500 ppm
Sep 27 03:31:51 dist-test-slave-j0hf krb5kdc[17376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:31:51.069054 17617 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:31:51.069458 17617 webserver.cc:460] Webserver started at http://127.0.98.2:43729/ using document root <none> and password file <none>
I20260927 03:31:51.069635 17617 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:51.069675 17617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:51.069761 17617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:31:51.070804 17617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-1/data/instance:
uuid: "ff587ca8567644ceaa8a515867d10d5d"
format_stamp: "Formatted at 2026-09-27 03:31:51 on dist-test-slave-j0hf"
server_key: "33acc06bc6406ff2614746a2ec8c70c2"
server_key_iv: "b7db06db214255bdb7cb1865bb5e34fa"
server_key_version: "encryptionkey@0"
I20260927 03:31:51.071098 17617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ff587ca8567644ceaa8a515867d10d5d"
format_stamp: "Formatted at 2026-09-27 03:31:51 on dist-test-slave-j0hf"
server_key: "33acc06bc6406ff2614746a2ec8c70c2"
server_key_iv: "b7db06db214255bdb7cb1865bb5e34fa"
server_key_version: "encryptionkey@0"
I20260927 03:31:51.072384 17617 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:31:51.073050 17632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:51.073225 17617 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:31:51.073310 17617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "ff587ca8567644ceaa8a515867d10d5d"
format_stamp: "Formatted at 2026-09-27 03:31:51 on dist-test-slave-j0hf"
server_key: "33acc06bc6406ff2614746a2ec8c70c2"
server_key_iv: "b7db06db214255bdb7cb1865bb5e34fa"
server_key_version: "encryptionkey@0"
I20260927 03:31:51.073385 17617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:51.101388 17617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:51.103248 17617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:51.103408 17617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:51.103685 17617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:31:51.104010 17617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:31:51.104065 17617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:51.104107 17617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:31:51.104135 17617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:51.109926 17617 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43797
I20260927 03:31:51.109983 17745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43797 every 8 connection(s)
I20260927 03:31:51.110292 17617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:31:51.110801 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 17617
I20260927 03:31:51.110919 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:31:51.111160 392 external_mini_cluster.cc:1428] Setting key 1986ea41ec6a45d84b6d6c88c6a65ae8
Sep 27 03:31:51 dist-test-slave-j0hf krb5kdc[17376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 03:31:51.120970 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:51.111029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35175 (local address 127.0.98.62:40219)
0927 03:31:51.111110 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:51.111112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:51.111324 (+ 212us) server_negotiation.cc:407] Connection header received
0927 03:31:51.112433 (+ 1109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:51.112434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:51.112447 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:51.112487 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:51.113205 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.113716 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.114898 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.115242 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.115319 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:51.117794 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:51.117796 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:51.117796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:31:51.117805 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:51.119467 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:51.119905 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:51.119908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:51.119908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:51.119998 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:51.120233 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:51.120233 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:51.120234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:51.120385 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:51.120452 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:51.120748 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 03:31:51.120867 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 03:31:51.121116 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:51.111004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40219 (local address 127.0.98.2:35175)
0927 03:31:51.111197 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:51.111241 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:51.112323 (+ 1082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:51.112584 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:51.112587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:51.112854 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:31:51.113089 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:51.113096 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.113815 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.113816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:51.114807 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:51.114809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.115341 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.115342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:51.115505 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:51.116058 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:31:51.116067 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:31:51.117691 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:31:51.119587 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:31:51.119594 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:31:51.119597 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:31:51.119832 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:31:51.120070 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:31:51.120071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:31:51.120071 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:31:51.120170 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:31:51.120449 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:31:51.120456 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:31:51.120608 (+ 152us) client_negotiation.cc:769] Sending connection context
0927 03:31:51.120717 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 03:31:51.120883 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":82,"threads_started":1}
I20260927 03:31:51.121562 17746 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40219
I20260927 03:31:51.121691 17746 heartbeater.cc:464] Registering TS with master...
I20260927 03:31:51.121887 17746 heartbeater.cc:511] Master 127.0.98.62:40219 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 03:31:51.122321 17421 ts_manager.cc:194] Registered new tserver with Master: ff587ca8567644ceaa8a515867d10d5d (127.0.98.2:43797)
I20260927 03:31:51.122818 17421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:31:51.136488 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40219
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:31:51.215597 17751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:51.215792 17751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:51.215816 17751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:51.216873 17751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:51.216904 17751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:51.216930 17751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:31:51.218286 17751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:51.218464 17751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:51.218623 17751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:31:51.223374 17759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:51.223398 17757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:51.223621 17751 server_base.cc:1034] running on GCE node
W20260927 03:31:51.223641 17756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:51.223945 17751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:51.224202 17751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:51.225361 17751 hybrid_clock.cc:648] HybridClock initialized: now 1790479911225352 us; error 26 us; skew 500 ppm
Sep 27 03:31:51 dist-test-slave-j0hf krb5kdc[17376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:31:51.227923 17751 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:31:51.228374 17751 webserver.cc:460] Webserver started at http://127.0.98.3:39927/ using document root <none> and password file <none>
I20260927 03:31:51.228610 17751 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:51.228679 17751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:51.228820 17751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:31:51.229848 17751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-2/data/instance:
uuid: "5cb422160e174ea0a4205d9de4d59dbb"
format_stamp: "Formatted at 2026-09-27 03:31:51 on dist-test-slave-j0hf"
server_key: "ff4834575d233053f17fb63e8a109140"
server_key_iv: "23c175c41dfa78841a92307d1adc5c6c"
server_key_version: "encryptionkey@0"
I20260927 03:31:51.230211 17751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5cb422160e174ea0a4205d9de4d59dbb"
format_stamp: "Formatted at 2026-09-27 03:31:51 on dist-test-slave-j0hf"
server_key: "ff4834575d233053f17fb63e8a109140"
server_key_iv: "23c175c41dfa78841a92307d1adc5c6c"
server_key_version: "encryptionkey@0"
I20260927 03:31:51.231649 17751 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:31:51.232414 17766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:51.232570 17751 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:31:51.232674 17751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "5cb422160e174ea0a4205d9de4d59dbb"
format_stamp: "Formatted at 2026-09-27 03:31:51 on dist-test-slave-j0hf"
server_key: "ff4834575d233053f17fb63e8a109140"
server_key_iv: "23c175c41dfa78841a92307d1adc5c6c"
server_key_version: "encryptionkey@0"
I20260927 03:31:51.232748 17751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:51.249634 17751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:51.252177 17751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:51.252341 17751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:51.252597 17751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:31:51.252912 17751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:31:51.252955 17751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:51.252978 17751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:31:51.252991 17751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:51.258646 17751 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33659
I20260927 03:31:51.258715 17879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33659 every 8 connection(s)
I20260927 03:31:51.259054 17751 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:31:51.261293 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 17751
I20260927 03:31:51.261410 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:31:51.261576 392 external_mini_cluster.cc:1428] Setting key d5621e7d77091a79db559c14a03abb6a
Sep 27 03:31:51 dist-test-slave-j0hf krb5kdc[17376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:31:51.268623 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:51.259894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46059 (local address 127.0.98.62:40219)
0927 03:31:51.260125 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:51.260127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:51.260159 (+ 32us) server_negotiation.cc:407] Connection header received
0927 03:31:51.261230 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:51.261231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:51.261246 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:51.261329 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:51.262218 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.262811 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.263972 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.264156 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.264210 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:51.266240 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:51.266241 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:51.266241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:31:51.266248 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:51.267513 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:51.267871 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:51.267871 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:51.267872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:51.267922 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:51.268125 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:51.268126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:51.268126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:51.268224 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:51.268272 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:51.268499 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 03:31:51.268559 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206}
I20260927 03:31:51.268846 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:51.259899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40219 (local address 127.0.98.3:46059)
0927 03:31:51.260057 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:51.260094 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:51.261093 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:51.261432 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:51.261437 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:51.261829 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:31:51.262131 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:51.262139 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.262920 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.262921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:51.263893 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:51.263896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.264230 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.264231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:51.264342 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:51.264834 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:31:51.264842 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:31:51.266157 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:31:51.267582 (+ 1425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:31:51.267588 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:31:51.267589 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:31:51.267806 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:31:51.267977 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:31:51.267978 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:31:51.267978 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:31:51.268073 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:31:51.268276 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:31:51.268280 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:31:51.268393 (+ 113us) client_negotiation.cc:769] Sending connection context
0927 03:31:51.268474 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 03:31:51.268591 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
I20260927 03:31:51.269424 17880 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40219
I20260927 03:31:51.269529 17880 heartbeater.cc:464] Registering TS with master...
I20260927 03:31:51.269675 17880 heartbeater.cc:511] Master 127.0.98.62:40219 requested a full tablet report, sending...
I20260927 03:31:51.270059 17421 ts_manager.cc:194] Registered new tserver with Master: 5cb422160e174ea0a4205d9de4d59dbb (127.0.98.3:33659)
I20260927 03:31:51.270447 17421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46059
I20260927 03:31:51.273162 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:31:51.281248 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:51.274716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44806 (local address 127.0.98.62:40219)
0927 03:31:51.274782 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:51.274784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:51.274794 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:31:51.274817 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:51.274818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:51.274829 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:51.274894 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:51.275705 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.276402 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.277301 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.277658 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.277700 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:51.278951 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:51.278952 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:51.278952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:31:51.278960 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:51.280254 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:51.280614 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:51.280615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:51.280616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:51.280668 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:51.280804 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:51.280804 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:51.280805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:51.280954 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:51.281029 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:51.281121 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 03:31:51.281182 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:31:51.282858 17421 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:31:51.284579 17421 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:31:51.290455 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:51.286719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37787 (local address 127.0.0.1:55898)
0927 03:31:51.286884 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:51.286926 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:51.286989 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:51.287242 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:51.287245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:51.287247 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:51.287319 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:51.287322 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.288389 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.288390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:51.289471 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:51.289474 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.290179 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.290180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:51.290261 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:51.290274 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 03:31:51.290339 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 03:31:51.290392 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":60,"threads_started":1}
I20260927 03:31:51.290722 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:51.286826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55898 (local address 127.0.98.1:37787)
0927 03:31:51.287034 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:51.287039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:51.287052 (+ 13us) server_negotiation.cc:407] Connection header received
0927 03:31:51.287097 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:51.287099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:51.287115 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:51.287236 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:51.287399 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.288280 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.289542 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.290092 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.290179 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:51.290240 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:51.290581 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 03:31:51.290637 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":122,"threads_started":1}
I20260927 03:31:51.291016 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:51.286735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43797 (local address 127.0.0.1:43136)
0927 03:31:51.287150 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:51.287158 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:51.287193 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:51.287593 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:51.287594 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:51.287596 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:51.287636 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:51.287638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.288934 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.288935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:51.289891 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:51.289893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.290803 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.290804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:51.290865 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:51.290869 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:31:51.290897 (+ 28us) client_negotiation.cc:240] Negotiation successful
0927 03:31:51.290932 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"spinlock_wait_cycles":896,"thread_start_us":59,"threads_started":1}
I20260927 03:31:51.291332 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:51.286988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43136 (local address 127.0.98.2:43797)
0927 03:31:51.287172 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:51.287176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:51.287189 (+ 13us) server_negotiation.cc:407] Connection header received
0927 03:31:51.287475 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:51.287478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:51.287494 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:51.287705 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:51.287785 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.288824 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.289988 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.290704 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.291122 (+ 418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:51.291193 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:51.291219 (+ 26us) server_negotiation.cc:299] Negotiation successful
0927 03:31:51.291256 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":101,"threads_started":1}
I20260927 03:31:51.291472 17546 tablet_service.cc:1467] Processing CreateTablet for tablet d37df54bad37446693f57f0453008ac5 (DEFAULT_TABLE table=test-table [id=2c0ac254260e4dc894e4078dd9ebaa90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:31:51.291882 17546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37df54bad37446693f57f0453008ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:31:51.291960 17680 tablet_service.cc:1467] Processing CreateTablet for tablet d37df54bad37446693f57f0453008ac5 (DEFAULT_TABLE table=test-table [id=2c0ac254260e4dc894e4078dd9ebaa90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:31:51.292279 17680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37df54bad37446693f57f0453008ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:31:51.294394 17894 tablet_bootstrap.cc:492] T d37df54bad37446693f57f0453008ac5 P 1ba160786a3c45fbbba1cb74544c37d1: Bootstrap starting.
I20260927 03:31:51.294636 17895 tablet_bootstrap.cc:492] T d37df54bad37446693f57f0453008ac5 P ff587ca8567644ceaa8a515867d10d5d: Bootstrap starting.
I20260927 03:31:51.295370 17895 tablet_bootstrap.cc:654] T d37df54bad37446693f57f0453008ac5 P ff587ca8567644ceaa8a515867d10d5d: Neither blocks nor log segments found. Creating new log.
I20260927 03:31:51.295370 17894 tablet_bootstrap.cc:654] T d37df54bad37446693f57f0453008ac5 P 1ba160786a3c45fbbba1cb74544c37d1: Neither blocks nor log segments found. Creating new log.
I20260927 03:31:51.295382 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:51.286733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33659 (local address 127.0.0.1:43436)
0927 03:31:51.287035 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:51.287044 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:51.287084 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:51.287537 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:51.287538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:51.287540 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:51.287587 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:51.287589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.289995 (+ 2406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.289996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:51.291053 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:51.291055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.295140 (+ 4085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.295141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:51.295212 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:51.295216 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:31:51.295267 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 03:31:51.295312 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":57,"threads_started":1}
I20260927 03:31:51.295449 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:51.287181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43436 (local address 127.0.98.3:33659)
0927 03:31:51.287348 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:51.287352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:51.287362 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:31:51.287399 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:51.287401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:51.287417 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:51.287660 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:51.287737 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.288823 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.291201 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.295035 (+ 3834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.295135 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:51.295193 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:51.295298 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 03:31:51.295338 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
I20260927 03:31:51.296000 17814 tablet_service.cc:1467] Processing CreateTablet for tablet d37df54bad37446693f57f0453008ac5 (DEFAULT_TABLE table=test-table [id=2c0ac254260e4dc894e4078dd9ebaa90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:31:51.296169 17894 log.cc:824] T d37df54bad37446693f57f0453008ac5 P 1ba160786a3c45fbbba1cb74544c37d1: Log is configured to *not* fsync() on all Append() calls
I20260927 03:31:51.296284 17895 log.cc:824] T d37df54bad37446693f57f0453008ac5 P ff587ca8567644ceaa8a515867d10d5d: Log is configured to *not* fsync() on all Append() calls
I20260927 03:31:51.296316 17814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37df54bad37446693f57f0453008ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:31:51.296944 17894 tablet_bootstrap.cc:492] T d37df54bad37446693f57f0453008ac5 P 1ba160786a3c45fbbba1cb74544c37d1: No bootstrap required, opened a new log
I20260927 03:31:51.296998 17894 ts_tablet_manager.cc:1397] T d37df54bad37446693f57f0453008ac5 P 1ba160786a3c45fbbba1cb74544c37d1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 03:31:51.298527 17894 raft_consensus.cc:348] T d37df54bad37446693f57f0453008ac5 P 1ba160786a3c45fbbba1cb74544c37d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba160786a3c45fbbba1cb74544c37d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37787 } } peers { permanent_uuid: "5cb422160e174ea0a4205d9de4d59dbb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33659 } } peers { permanent_uuid: "ff587ca8567644ceaa8a515867d10d5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43797 } }
I20260927 03:31:51.298660 17894 raft_consensus.cc:374] T d37df54bad37446693f57f0453008ac5 P 1ba160786a3c45fbbba1cb74544c37d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:31:51.298682 17894 raft_consensus.cc:729] T d37df54bad37446693f57f0453008ac5 P 1ba160786a3c45fbbba1cb74544c37d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ba160786a3c45fbbba1cb74544c37d1, State: Initialized, Role: FOLLOWER
I20260927 03:31:51.298794 17894 consensus_queue.cc:260] T d37df54bad37446693f57f0453008ac5 P 1ba160786a3c45fbbba1cb74544c37d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba160786a3c45fbbba1cb74544c37d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37787 } } peers { permanent_uuid: "5cb422160e174ea0a4205d9de4d59dbb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33659 } } peers { permanent_uuid: "ff587ca8567644ceaa8a515867d10d5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43797 } }
I20260927 03:31:51.299072 17894 ts_tablet_manager.cc:1428] T d37df54bad37446693f57f0453008ac5 P 1ba160786a3c45fbbba1cb74544c37d1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:31:51.299095 17612 heartbeater.cc:503] Master 127.0.98.62:40219 was elected leader, sending a full tablet report...
I20260927 03:31:51.299490 17895 tablet_bootstrap.cc:492] T d37df54bad37446693f57f0453008ac5 P ff587ca8567644ceaa8a515867d10d5d: No bootstrap required, opened a new log
I20260927 03:31:51.299561 17895 ts_tablet_manager.cc:1397] T d37df54bad37446693f57f0453008ac5 P ff587ca8567644ceaa8a515867d10d5d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 03:31:51.299587 17899 tablet_bootstrap.cc:492] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb: Bootstrap starting.
I20260927 03:31:51.300241 17899 tablet_bootstrap.cc:654] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb: Neither blocks nor log segments found. Creating new log.
I20260927 03:31:51.300724 17899 log.cc:824] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb: Log is configured to *not* fsync() on all Append() calls
I20260927 03:31:51.301009 17895 raft_consensus.cc:348] T d37df54bad37446693f57f0453008ac5 P ff587ca8567644ceaa8a515867d10d5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba160786a3c45fbbba1cb74544c37d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37787 } } peers { permanent_uuid: "5cb422160e174ea0a4205d9de4d59dbb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33659 } } peers { permanent_uuid: "ff587ca8567644ceaa8a515867d10d5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43797 } }
I20260927 03:31:51.301124 17895 raft_consensus.cc:374] T d37df54bad37446693f57f0453008ac5 P ff587ca8567644ceaa8a515867d10d5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:31:51.301146 17895 raft_consensus.cc:729] T d37df54bad37446693f57f0453008ac5 P ff587ca8567644ceaa8a515867d10d5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff587ca8567644ceaa8a515867d10d5d, State: Initialized, Role: FOLLOWER
I20260927 03:31:51.301223 17895 consensus_queue.cc:260] T d37df54bad37446693f57f0453008ac5 P ff587ca8567644ceaa8a515867d10d5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba160786a3c45fbbba1cb74544c37d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37787 } } peers { permanent_uuid: "5cb422160e174ea0a4205d9de4d59dbb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33659 } } peers { permanent_uuid: "ff587ca8567644ceaa8a515867d10d5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43797 } }
I20260927 03:31:51.301398 17899 tablet_bootstrap.cc:492] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb: No bootstrap required, opened a new log
I20260927 03:31:51.301453 17895 ts_tablet_manager.cc:1428] T d37df54bad37446693f57f0453008ac5 P ff587ca8567644ceaa8a515867d10d5d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:31:51.301465 17899 ts_tablet_manager.cc:1397] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:31:51.301496 17746 heartbeater.cc:503] Master 127.0.98.62:40219 was elected leader, sending a full tablet report...
I20260927 03:31:51.303128 17899 raft_consensus.cc:348] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba160786a3c45fbbba1cb74544c37d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37787 } } peers { permanent_uuid: "5cb422160e174ea0a4205d9de4d59dbb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33659 } } peers { permanent_uuid: "ff587ca8567644ceaa8a515867d10d5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43797 } }
I20260927 03:31:51.303249 17899 raft_consensus.cc:374] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:31:51.303270 17899 raft_consensus.cc:729] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb422160e174ea0a4205d9de4d59dbb, State: Initialized, Role: FOLLOWER
I20260927 03:31:51.303340 17899 consensus_queue.cc:260] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba160786a3c45fbbba1cb74544c37d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37787 } } peers { permanent_uuid: "5cb422160e174ea0a4205d9de4d59dbb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33659 } } peers { permanent_uuid: "ff587ca8567644ceaa8a515867d10d5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43797 } }
I20260927 03:31:51.303601 17899 ts_tablet_manager.cc:1428] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 03:31:51.303637 17880 heartbeater.cc:503] Master 127.0.98.62:40219 was elected leader, sending a full tablet report...
I20260927 03:31:51.333689 17902 raft_consensus.cc:482] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:31:51.333834 17902 raft_consensus.cc:504] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba160786a3c45fbbba1cb74544c37d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37787 } } peers { permanent_uuid: "5cb422160e174ea0a4205d9de4d59dbb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33659 } } peers { permanent_uuid: "ff587ca8567644ceaa8a515867d10d5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43797 } }
I20260927 03:31:51.334323 17902 leader_election.cc:288] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ba160786a3c45fbbba1cb74544c37d1 (127.0.98.1:37787), ff587ca8567644ceaa8a515867d10d5d (127.0.98.2:43797)
I20260927 03:31:51.337513 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:51.334566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48035 (local address 127.0.98.2:43797)
0927 03:31:51.334608 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:51.334610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:51.334619 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:31:51.334724 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:51.334726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:51.334741 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:51.334808 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:51.334987 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.335665 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.336778 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.337164 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.337252 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:51.337262 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:51.337412 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 03:31:51.337457 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:31:51.337510 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:51.334506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43797 (local address 127.0.98.3:48035)
0927 03:31:51.334559 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:51.334571 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:51.334613 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:51.334822 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:51.334823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:51.334824 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:51.334902 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:51.334904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.335752 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.335753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:51.336697 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:51.336699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.337252 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.337253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:51.337332 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:51.337335 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:31:51.337393 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 03:31:51.337432 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":4}
I20260927 03:31:51.337868 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:51.334588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58273 (local address 127.0.98.1:37787)
0927 03:31:51.334639 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:51.334642 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:51.334959 (+ 317us) server_negotiation.cc:407] Connection header received
0927 03:31:51.335045 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:51.335046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:51.335060 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:51.335144 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:51.335373 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.335950 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.337081 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.337452 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.337535 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:51.337545 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:51.337744 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 03:31:51.337794 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 03:31:51.337949 17700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37df54bad37446693f57f0453008ac5" candidate_uuid: "5cb422160e174ea0a4205d9de4d59dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff587ca8567644ceaa8a515867d10d5d" is_pre_election: true
I20260927 03:31:51.338115 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:51.334546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37787 (local address 127.0.98.3:58273)
0927 03:31:51.334887 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:51.334896 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:51.334970 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:51.335149 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:51.335150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:51.335151 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:51.335240 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:51.335242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.336048 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.336049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:51.336987 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:51.336989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.337568 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.337569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:51.337643 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:51.337648 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:31:51.337979 (+ 331us) client_negotiation.cc:240] Negotiation successful
0927 03:31:51.338025 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":117,"threads_started":1}
I20260927 03:31:51.338124 17700 raft_consensus.cc:2454] T d37df54bad37446693f57f0453008ac5 P ff587ca8567644ceaa8a515867d10d5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cb422160e174ea0a4205d9de4d59dbb in term 0.
I20260927 03:31:51.338418 17769 leader_election.cc:302] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb422160e174ea0a4205d9de4d59dbb, ff587ca8567644ceaa8a515867d10d5d; no voters:
I20260927 03:31:51.338436 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37df54bad37446693f57f0453008ac5" candidate_uuid: "5cb422160e174ea0a4205d9de4d59dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ba160786a3c45fbbba1cb74544c37d1" is_pre_election: true
I20260927 03:31:51.338579 17566 raft_consensus.cc:2454] T d37df54bad37446693f57f0453008ac5 P 1ba160786a3c45fbbba1cb74544c37d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cb422160e174ea0a4205d9de4d59dbb in term 0.
I20260927 03:31:51.338569 17902 raft_consensus.cc:2781] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:31:51.338627 17902 raft_consensus.cc:482] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:31:51.338653 17902 raft_consensus.cc:3037] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:31:51.339510 17902 raft_consensus.cc:504] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba160786a3c45fbbba1cb74544c37d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37787 } } peers { permanent_uuid: "5cb422160e174ea0a4205d9de4d59dbb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33659 } } peers { permanent_uuid: "ff587ca8567644ceaa8a515867d10d5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43797 } }
I20260927 03:31:51.339658 17902 leader_election.cc:288] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb [CANDIDATE]: Term 1 election: Requested vote from peers 1ba160786a3c45fbbba1cb74544c37d1 (127.0.98.1:37787), ff587ca8567644ceaa8a515867d10d5d (127.0.98.2:43797)
I20260927 03:31:51.339782 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37df54bad37446693f57f0453008ac5" candidate_uuid: "5cb422160e174ea0a4205d9de4d59dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ba160786a3c45fbbba1cb74544c37d1"
I20260927 03:31:51.339833 17700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37df54bad37446693f57f0453008ac5" candidate_uuid: "5cb422160e174ea0a4205d9de4d59dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff587ca8567644ceaa8a515867d10d5d"
I20260927 03:31:51.339885 17566 raft_consensus.cc:3037] T d37df54bad37446693f57f0453008ac5 P 1ba160786a3c45fbbba1cb74544c37d1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:31:51.339934 17700 raft_consensus.cc:3037] T d37df54bad37446693f57f0453008ac5 P ff587ca8567644ceaa8a515867d10d5d [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:31:51.340602 17566 raft_consensus.cc:2454] T d37df54bad37446693f57f0453008ac5 P 1ba160786a3c45fbbba1cb74544c37d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb422160e174ea0a4205d9de4d59dbb in term 1.
I20260927 03:31:51.340768 17700 raft_consensus.cc:2454] T d37df54bad37446693f57f0453008ac5 P ff587ca8567644ceaa8a515867d10d5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cb422160e174ea0a4205d9de4d59dbb in term 1.
I20260927 03:31:51.340813 17770 leader_election.cc:302] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ba160786a3c45fbbba1cb74544c37d1, 5cb422160e174ea0a4205d9de4d59dbb; no voters:
I20260927 03:31:51.340919 17902 raft_consensus.cc:2781] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:31:51.341063 17902 raft_consensus.cc:686] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb [term 1 LEADER]: Becoming Leader. State: Replica: 5cb422160e174ea0a4205d9de4d59dbb, State: Running, Role: LEADER
I20260927 03:31:51.341164 17902 consensus_queue.cc:237] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba160786a3c45fbbba1cb74544c37d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37787 } } peers { permanent_uuid: "5cb422160e174ea0a4205d9de4d59dbb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33659 } } peers { permanent_uuid: "ff587ca8567644ceaa8a515867d10d5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43797 } }
I20260927 03:31:51.342038 17421 catalog_manager.cc:5495] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cb422160e174ea0a4205d9de4d59dbb (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5cb422160e174ea0a4205d9de4d59dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ba160786a3c45fbbba1cb74544c37d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb422160e174ea0a4205d9de4d59dbb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff587ca8567644ceaa8a515867d10d5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43797 } health_report { overall_health: UNKNOWN } } }
I20260927 03:31:51.348202 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:51.345997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43452 (local address 127.0.98.3:33659)
0927 03:31:51.346053 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:51.346055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:51.346088 (+ 33us) server_negotiation.cc:407] Connection header received
0927 03:31:51.346114 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:51.346114 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:51.346129 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:51.346179 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:31:51.346262 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.346705 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.347497 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.347694 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.347782 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:51.347892 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:31:51.347988 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:31:51.348054 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:51.348084 (+ 30us) server_negotiation.cc:299] Negotiation successful
0927 03:31:51.348102 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 03:31:51.349915 17566 raft_consensus.cc:1261] T d37df54bad37446693f57f0453008ac5 P 1ba160786a3c45fbbba1cb74544c37d1 [term 1 FOLLOWER]: Refusing update from remote peer 5cb422160e174ea0a4205d9de4d59dbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:31:51.349915 17700 raft_consensus.cc:1261] T d37df54bad37446693f57f0453008ac5 P ff587ca8567644ceaa8a515867d10d5d [term 1 FOLLOWER]: Refusing update from remote peer 5cb422160e174ea0a4205d9de4d59dbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:31:51.350298 17904 consensus_queue.cc:1035] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ba160786a3c45fbbba1cb74544c37d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:31:51.350417 17902 consensus_queue.cc:1035] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff587ca8567644ceaa8a515867d10d5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:31:51.352821 17907 mvcc.cc:205] Tried to move back new op lower bound from 7333805716886614016 to 7333805716854194176. Current Snapshot: MvccSnapshot[applied={T|T < 7333805716886614016}]
I20260927 03:31:51.353258 17909 mvcc.cc:205] Tried to move back new op lower bound from 7333805716886614016 to 7333805716854194176. Current Snapshot: MvccSnapshot[applied={T|T < 7333805716886614016}]
I20260927 03:31:51.353067 17908 mvcc.cc:205] Tried to move back new op lower bound from 7333805716886614016 to 7333805716854194176. Current Snapshot: MvccSnapshot[applied={T|T < 7333805716886614016}]
I20260927 03:31:51.355574 17420 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:31:51.355674 17420 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:31:51.356465 17420 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c3155d3c1d074ff48a10bbf72beb1a44: Sending DeleteTablet for 3 replicas of tablet d37df54bad37446693f57f0453008ac5
I20260927 03:31:51.356786 17814 tablet_service.cc:1514] Processing DeleteTablet for tablet d37df54bad37446693f57f0453008ac5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:31:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43436
I20260927 03:31:51.356792 17546 tablet_service.cc:1514] Processing DeleteTablet for tablet d37df54bad37446693f57f0453008ac5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:31:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55898
I20260927 03:31:51.356848 17680 tablet_service.cc:1514] Processing DeleteTablet for tablet d37df54bad37446693f57f0453008ac5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:31:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43136
I20260927 03:31:51.357059 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 17483
I20260927 03:31:51.358610 17919 tablet_replica.cc:330] T d37df54bad37446693f57f0453008ac5 P ff587ca8567644ceaa8a515867d10d5d: stopping tablet replica
I20260927 03:31:51.358732 17919 raft_consensus.cc:2229] T d37df54bad37446693f57f0453008ac5 P ff587ca8567644ceaa8a515867d10d5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:31:51.358853 17919 raft_consensus.cc:2258] T d37df54bad37446693f57f0453008ac5 P ff587ca8567644ceaa8a515867d10d5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:31:51.359736 17919 ts_tablet_manager.cc:1905] T d37df54bad37446693f57f0453008ac5 P ff587ca8567644ceaa8a515867d10d5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:31:51.361928 17919 ts_tablet_manager.cc:1918] T d37df54bad37446693f57f0453008ac5 P ff587ca8567644ceaa8a515867d10d5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:31:51.362038 17919 log.cc:1196] T d37df54bad37446693f57f0453008ac5 P ff587ca8567644ceaa8a515867d10d5d: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-1/wal/wals/d37df54bad37446693f57f0453008ac5
I20260927 03:31:51.362329 17919 ts_tablet_manager.cc:1939] T d37df54bad37446693f57f0453008ac5 P ff587ca8567644ceaa8a515867d10d5d: Deleting consensus metadata
I20260927 03:31:51.362792 17407 catalog_manager.cc:4841] TS ff587ca8567644ceaa8a515867d10d5d (127.0.98.2:43797): tablet d37df54bad37446693f57f0453008ac5 (table test-table [id=2c0ac254260e4dc894e4078dd9ebaa90]) successfully deleted
W20260927 03:31:51.363595 17408 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37787 (error 108)
I20260927 03:31:51.364104 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:51.363965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37787 (local address 127.0.0.1:55906)
0927 03:31:51.364030 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:51.364069 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 03:31:51.364172 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 17617
I20260927 03:31:51.364936 17917 tablet_replica.cc:330] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb: stopping tablet replica
I20260927 03:31:51.365068 17917 raft_consensus.cc:2229] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:31:51.365209 17917 raft_consensus.cc:2258] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:31:51.365590 17917 ts_tablet_manager.cc:1905] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:31:51.366932 17917 ts_tablet_manager.cc:1918] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:31:51.367022 17917 log.cc:1196] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790479675865103-392-0/minicluster-data/ts-2/wal/wals/d37df54bad37446693f57f0453008ac5
I20260927 03:31:51.367316 17917 ts_tablet_manager.cc:1939] T d37df54bad37446693f57f0453008ac5 P 5cb422160e174ea0a4205d9de4d59dbb: Deleting consensus metadata
I20260927 03:31:51.367746 17405 catalog_manager.cc:4841] TS 5cb422160e174ea0a4205d9de4d59dbb (127.0.98.3:33659): tablet d37df54bad37446693f57f0453008ac5 (table test-table [id=2c0ac254260e4dc894e4078dd9ebaa90]) successfully deleted
W20260927 03:31:51.368073 17408 catalog_manager.cc:4584] TS 1ba160786a3c45fbbba1cb74544c37d1 (127.0.98.1:37787): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d37df54bad37446693f57f0453008ac5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37787: connect: Connection refused (error 111)
I20260927 03:31:51.371220 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 17751
I20260927 03:31:51.376361 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 17389
2026-09-27T03:31:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1906 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T03:31:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:31:51Z Disabled control of system clock
I20260927 03:31:51.403925 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33857
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 03:31:51.502244 17924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:51.502586 17924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:51.502671 17924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:51.504858 17924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:31:51.504920 17924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:51.504941 17924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:51.504961 17924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:31:51.504987 17924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:31:51.506390 17924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33857
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:51.506595 17924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:51.506829 17924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:31:51.511483 17932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:51.511485 17929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:51.511492 17930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:51.511849 17924 server_base.cc:1034] running on GCE node
I20260927 03:31:51.512060 17924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:51.512282 17924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:51.513424 17924 hybrid_clock.cc:648] HybridClock initialized: now 1790479911513399 us; error 37 us; skew 500 ppm
I20260927 03:31:51.514506 17924 webserver.cc:460] Webserver started at http://127.0.98.62:34743/ using document root <none> and password file <none>
I20260927 03:31:51.514690 17924 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:51.514729 17924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:51.514815 17924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:31:51.515880 17924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "5e7dfef021d24f0dbed7379047dc77df"
format_stamp: "Formatted at 2026-09-27 03:31:51 on dist-test-slave-j0hf"
server_key: "0725d1617ca854a5dc02bfd321bfbce6"
server_key_iv: "40fc011cabf50e51062b2cc6ddb18d8d"
server_key_version: "encryptionkey@0"
I20260927 03:31:51.516177 17924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "5e7dfef021d24f0dbed7379047dc77df"
format_stamp: "Formatted at 2026-09-27 03:31:51 on dist-test-slave-j0hf"
server_key: "0725d1617ca854a5dc02bfd321bfbce6"
server_key_iv: "40fc011cabf50e51062b2cc6ddb18d8d"
server_key_version: "encryptionkey@0"
I20260927 03:31:51.517311 17924 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:31:51.517937 17938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:51.518146 17924 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:31:51.518215 17924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "5e7dfef021d24f0dbed7379047dc77df"
format_stamp: "Formatted at 2026-09-27 03:31:51 on dist-test-slave-j0hf"
server_key: "0725d1617ca854a5dc02bfd321bfbce6"
server_key_iv: "40fc011cabf50e51062b2cc6ddb18d8d"
server_key_version: "encryptionkey@0"
I20260927 03:31:51.518306 17924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:51.529726 17924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:51.529999 17924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:51.530138 17924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:51.534097 17924 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33857
I20260927 03:31:51.534129 17990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33857 every 8 connection(s)
I20260927 03:31:51.534494 17924 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:31:51.535459 17991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:31:51.537724 17991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e7dfef021d24f0dbed7379047dc77df: Bootstrap starting.
I20260927 03:31:51.538311 17991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e7dfef021d24f0dbed7379047dc77df: Neither blocks nor log segments found. Creating new log.
I20260927 03:31:51.538348 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 17924
I20260927 03:31:51.538476 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:31:51.538661 392 external_mini_cluster.cc:1428] Setting key 2d0ffb4b56827e8ff62895f90b9596cc
I20260927 03:31:51.538650 17991 log.cc:824] T 00000000000000000000000000000000 P 5e7dfef021d24f0dbed7379047dc77df: Log is configured to *not* fsync() on all Append() calls
I20260927 03:31:51.539398 17991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e7dfef021d24f0dbed7379047dc77df: No bootstrap required, opened a new log
I20260927 03:31:51.541481 17991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e7dfef021d24f0dbed7379047dc77df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7dfef021d24f0dbed7379047dc77df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33857 } }
I20260927 03:31:51.541641 17991 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e7dfef021d24f0dbed7379047dc77df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:31:51.541687 17991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e7dfef021d24f0dbed7379047dc77df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e7dfef021d24f0dbed7379047dc77df, State: Initialized, Role: FOLLOWER
I20260927 03:31:51.541811 17991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e7dfef021d24f0dbed7379047dc77df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7dfef021d24f0dbed7379047dc77df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33857 } }
I20260927 03:31:51.541906 17991 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e7dfef021d24f0dbed7379047dc77df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:31:51.541953 17991 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e7dfef021d24f0dbed7379047dc77df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:31:51.542001 17991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e7dfef021d24f0dbed7379047dc77df [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:31:51.542515 17991 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e7dfef021d24f0dbed7379047dc77df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7dfef021d24f0dbed7379047dc77df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33857 } }
I20260927 03:31:51.542644 17991 leader_election.cc:302] T 00000000000000000000000000000000 P 5e7dfef021d24f0dbed7379047dc77df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e7dfef021d24f0dbed7379047dc77df; no voters:
I20260927 03:31:51.542830 17991 leader_election.cc:288] T 00000000000000000000000000000000 P 5e7dfef021d24f0dbed7379047dc77df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:31:51.542874 17996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e7dfef021d24f0dbed7379047dc77df [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:31:51.543020 17996 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e7dfef021d24f0dbed7379047dc77df [term 1 LEADER]: Becoming Leader. State: Replica: 5e7dfef021d24f0dbed7379047dc77df, State: Running, Role: LEADER
I20260927 03:31:51.543144 17991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e7dfef021d24f0dbed7379047dc77df [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:31:51.543177 17996 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e7dfef021d24f0dbed7379047dc77df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7dfef021d24f0dbed7379047dc77df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33857 } }
I20260927 03:31:51.543620 17998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7dfef021d24f0dbed7379047dc77df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e7dfef021d24f0dbed7379047dc77df. Latest consensus state: current_term: 1 leader_uuid: "5e7dfef021d24f0dbed7379047dc77df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7dfef021d24f0dbed7379047dc77df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33857 } } }
I20260927 03:31:51.543780 17998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7dfef021d24f0dbed7379047dc77df [sys.catalog]: This master's current role is: LEADER
I20260927 03:31:51.544068 18004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:31:51.544607 18004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:31:51.544799 17995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:51.539186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58636 (local address 127.0.98.62:33857)
0927 03:31:51.539398 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:51.539407 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:51.539465 (+ 58us) server_negotiation.cc:407] Connection header received
0927 03:31:51.539922 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:51.539929 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:51.539989 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:51.540138 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:51.540373 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.541256 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.542290 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.542788 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.542930 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:51.543853 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:51.543857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:51.543858 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:31:51.544070 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:51.544118 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:31:51.544133 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:51.544194 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:51.544489 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 03:31:51.544573 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
I20260927 03:31:51.543725 17997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7dfef021d24f0dbed7379047dc77df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e7dfef021d24f0dbed7379047dc77df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7dfef021d24f0dbed7379047dc77df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33857 } } }
I20260927 03:31:51.545168 17997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7dfef021d24f0dbed7379047dc77df [sys.catalog]: This master's current role is: LEADER
I20260927 03:31:51.546591 18004 catalog_manager.cc:1347] Generated new cluster ID: 7e6620ca65fe4d44aeb04cb02c5b7ea4
I20260927 03:31:51.546646 18004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:31:51.564409 18004 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:31:51.564989 18004 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:31:51.571288 18004 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e7dfef021d24f0dbed7379047dc77df: Generated new TSK 0
I20260927 03:31:51.571516 18004 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:31:51.573797 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33857
--builtin_ntp_servers=127.0.98.20:38811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 03:31:51.692147 18015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:51.692472 18015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:51.692514 18015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:51.694190 18015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:51.694252 18015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:51.694293 18015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:31:51.696332 18015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:51.696599 18015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:51.696956 18015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:31:51.702206 18021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:51.702183 18020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:51.702280 18023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:51.702504 18015 server_base.cc:1034] running on GCE node
I20260927 03:31:51.702775 18015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:51.703022 18015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:51.704182 18015 hybrid_clock.cc:648] HybridClock initialized: now 1790479911704165 us; error 35 us; skew 500 ppm
I20260927 03:31:51.705431 18015 webserver.cc:460] Webserver started at http://127.0.98.1:33805/ using document root <none> and password file <none>
I20260927 03:31:51.705680 18015 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:51.705749 18015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:51.705886 18015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:31:51.706938 18015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-0/data/instance:
uuid: "049ba44da45346b788359a510ea6549b"
format_stamp: "Formatted at 2026-09-27 03:31:51 on dist-test-slave-j0hf"
server_key: "8787fbcc79fa886d6932aafdacf0fcf1"
server_key_iv: "7a89dc8744ddca4338699fed8d9708f6"
server_key_version: "encryptionkey@0"
I20260927 03:31:51.707298 18015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-0/wal/instance:
uuid: "049ba44da45346b788359a510ea6549b"
format_stamp: "Formatted at 2026-09-27 03:31:51 on dist-test-slave-j0hf"
server_key: "8787fbcc79fa886d6932aafdacf0fcf1"
server_key_iv: "7a89dc8744ddca4338699fed8d9708f6"
server_key_version: "encryptionkey@0"
I20260927 03:31:51.708676 18015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:31:51.709497 18029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:51.709726 18015 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:31:51.709827 18015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "049ba44da45346b788359a510ea6549b"
format_stamp: "Formatted at 2026-09-27 03:31:51 on dist-test-slave-j0hf"
server_key: "8787fbcc79fa886d6932aafdacf0fcf1"
server_key_iv: "7a89dc8744ddca4338699fed8d9708f6"
server_key_version: "encryptionkey@0"
I20260927 03:31:51.709899 18015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:51.736438 18015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:51.736721 18015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:51.736824 18015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:51.737020 18015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:31:51.737382 18015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:31:51.737418 18015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:51.737490 18015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:31:51.737519 18015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:51.743219 18015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41647
I20260927 03:31:51.743294 18142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41647 every 8 connection(s)
I20260927 03:31:51.743680 18015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:31:51.749197 17995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:51.744482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42229 (local address 127.0.98.62:33857)
0927 03:31:51.744526 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:51.744528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:51.744760 (+ 232us) server_negotiation.cc:407] Connection header received
0927 03:31:51.745383 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:51.745384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:51.745397 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:51.745453 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:51.746165 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.746687 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.747755 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.747966 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.748041 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:51.748866 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:51.748868 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:51.748868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:31:51.748874 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:51.748889 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:31:51.748893 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:51.748977 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:51.749075 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 03:31:51.749111 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:31:51.749454 18145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:51.744458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33857 (local address 127.0.98.1:42229)
0927 03:31:51.744638 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:51.744673 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:51.745265 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:51.745526 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:51.745531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:51.745541 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:31:51.746059 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:51.746065 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.746798 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.746799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:51.747651 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:51.747653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.748062 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.748063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:51.748206 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:51.748720 (+ 514us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:31:51.748720 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:31:51.748750 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:31:51.748773 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:31:51.748981 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:31:51.748987 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:31:51.748990 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:31:51.749078 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 03:31:51.749186 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260927 03:31:51.749904 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 18015
I20260927 03:31:51.750016 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:31:51.750053 18143 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33857
I20260927 03:31:51.750170 18143 heartbeater.cc:464] Registering TS with master...
I20260927 03:31:51.750247 392 external_mini_cluster.cc:1428] Setting key adadd1e653d0a247431880d786dad6db
I20260927 03:31:51.750399 18143 heartbeater.cc:511] Master 127.0.98.62:33857 requested a full tablet report, sending...
I20260927 03:31:51.750979 17954 ts_manager.cc:194] Registered new tserver with Master: 049ba44da45346b788359a510ea6549b (127.0.98.1:41647)
I20260927 03:31:51.751256 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33857
--builtin_ntp_servers=127.0.98.20:38811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 03:31:51.751745 17954 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:42229
W20260927 03:31:51.833501 18146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:51.833712 18146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:51.833734 18146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:51.834854 18146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:51.834890 18146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:51.834914 18146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:31:51.836319 18146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:51.836508 18146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:51.836750 18146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:31:51.841836 18152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:51.841926 18154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:51.841928 18151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:51.841987 18146 server_base.cc:1034] running on GCE node
I20260927 03:31:51.842370 18146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:51.842640 18146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:51.843811 18146 hybrid_clock.cc:648] HybridClock initialized: now 1790479911843782 us; error 37 us; skew 500 ppm
I20260927 03:31:51.845168 18146 webserver.cc:460] Webserver started at http://127.0.98.2:42151/ using document root <none> and password file <none>
I20260927 03:31:51.845423 18146 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:51.845506 18146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:51.845671 18146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:31:51.846746 18146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-1/data/instance:
uuid: "fa9cc2049c1b437ba32625e979cbd0a1"
format_stamp: "Formatted at 2026-09-27 03:31:51 on dist-test-slave-j0hf"
server_key: "673f7412283d8b9d24da5bef6857c9a5"
server_key_iv: "5cd250ec83f1fa60a787529dec2e3eb3"
server_key_version: "encryptionkey@0"
I20260927 03:31:51.847114 18146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fa9cc2049c1b437ba32625e979cbd0a1"
format_stamp: "Formatted at 2026-09-27 03:31:51 on dist-test-slave-j0hf"
server_key: "673f7412283d8b9d24da5bef6857c9a5"
server_key_iv: "5cd250ec83f1fa60a787529dec2e3eb3"
server_key_version: "encryptionkey@0"
I20260927 03:31:51.848462 18146 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:31:51.849159 18160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:51.849320 18146 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:31:51.849423 18146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "fa9cc2049c1b437ba32625e979cbd0a1"
format_stamp: "Formatted at 2026-09-27 03:31:51 on dist-test-slave-j0hf"
server_key: "673f7412283d8b9d24da5bef6857c9a5"
server_key_iv: "5cd250ec83f1fa60a787529dec2e3eb3"
server_key_version: "encryptionkey@0"
I20260927 03:31:51.849507 18146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:51.866487 18146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:51.866793 18146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:51.866940 18146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:51.867170 18146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:31:51.867563 18146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:31:51.867620 18146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:51.867671 18146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:31:51.867698 18146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:51.873379 18146 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40691
I20260927 03:31:51.873427 18273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40691 every 8 connection(s)
I20260927 03:31:51.873759 18146 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:31:51.876220 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 18146
I20260927 03:31:51.876327 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:31:51.876528 392 external_mini_cluster.cc:1428] Setting key 4d155e380217a1b70ef071c5427de38f
I20260927 03:31:51.877696 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33857
--builtin_ntp_servers=127.0.98.20:38811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 03:31:51.879722 17995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:51.874551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44249 (local address 127.0.98.62:33857)
0927 03:31:51.874606 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:51.874608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:51.874837 (+ 229us) server_negotiation.cc:407] Connection header received
0927 03:31:51.875303 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:51.875304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:51.875313 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:51.875371 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:51.876066 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.876824 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.878107 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.878423 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.878495 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:51.879324 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:51.879326 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:51.879327 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:31:51.879332 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:51.879349 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:31:51.879365 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:51.879489 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:51.879590 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 03:31:51.879619 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:31:51.879858 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:51.874523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33857 (local address 127.0.98.2:44249)
0927 03:31:51.874706 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:51.874747 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:51.875222 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:51.875448 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:51.875451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:51.875460 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:31:51.875976 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:51.875980 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.876925 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.876926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:51.878021 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:51.878024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:51.878504 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:51.878505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:51.878611 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:51.879187 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:31:51.879187 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:31:51.879217 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:31:51.879239 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:31:51.879506 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:31:51.879510 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:31:51.879513 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:31:51.879574 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 03:31:51.879654 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
I20260927 03:31:51.880321 18274 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33857
I20260927 03:31:51.880412 18274 heartbeater.cc:464] Registering TS with master...
I20260927 03:31:51.880609 18274 heartbeater.cc:511] Master 127.0.98.62:33857 requested a full tablet report, sending...
I20260927 03:31:51.880992 17954 ts_manager.cc:194] Registered new tserver with Master: fa9cc2049c1b437ba32625e979cbd0a1 (127.0.98.2:40691)
I20260927 03:31:51.881361 17954 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:44249
W20260927 03:31:51.961588 18277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:51.961812 18277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:51.961846 18277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:51.963063 18277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:51.963125 18277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:51.963167 18277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:31:51.964529 18277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:51.964823 18277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:51.965119 18277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:31:51.969937 18283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:51.969964 18285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:51.969950 18282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:51.970204 18277 server_base.cc:1034] running on GCE node
I20260927 03:31:51.970402 18277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:51.970628 18277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:51.971782 18277 hybrid_clock.cc:648] HybridClock initialized: now 1790479911971751 us; error 51 us; skew 500 ppm
I20260927 03:31:51.973171 18277 webserver.cc:460] Webserver started at http://127.0.98.3:40549/ using document root <none> and password file <none>
I20260927 03:31:51.973414 18277 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:51.973490 18277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:51.973639 18277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:31:51.974699 18277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-2/data/instance:
uuid: "d1f73f0025ed44c4869a6e70ae564984"
format_stamp: "Formatted at 2026-09-27 03:31:51 on dist-test-slave-j0hf"
server_key: "34b96c739d9d067671eb92e8b845df67"
server_key_iv: "48eddc1fcd4f6050c5c9c2af4b30ed2f"
server_key_version: "encryptionkey@0"
I20260927 03:31:51.975056 18277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d1f73f0025ed44c4869a6e70ae564984"
format_stamp: "Formatted at 2026-09-27 03:31:51 on dist-test-slave-j0hf"
server_key: "34b96c739d9d067671eb92e8b845df67"
server_key_iv: "48eddc1fcd4f6050c5c9c2af4b30ed2f"
server_key_version: "encryptionkey@0"
I20260927 03:31:51.976380 18277 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:31:51.977061 18291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:51.977224 18277 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:31:51.977329 18277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "d1f73f0025ed44c4869a6e70ae564984"
format_stamp: "Formatted at 2026-09-27 03:31:51 on dist-test-slave-j0hf"
server_key: "34b96c739d9d067671eb92e8b845df67"
server_key_iv: "48eddc1fcd4f6050c5c9c2af4b30ed2f"
server_key_version: "encryptionkey@0"
I20260927 03:31:51.977403 18277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:52.005836 18277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:52.006179 18277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:52.006342 18277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:52.006570 18277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:31:52.006929 18277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:31:52.006986 18277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:52.007040 18277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:31:52.007066 18277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:52.012693 18277 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36131
I20260927 03:31:52.012739 18404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36131 every 8 connection(s)
I20260927 03:31:52.013072 18277 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:31:52.013276 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 18277
I20260927 03:31:52.013401 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:31:52.013616 392 external_mini_cluster.cc:1428] Setting key 1e934659b7b72c5c5bc1b8c2926ff54d
I20260927 03:31:52.018823 17995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:52.014060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52627 (local address 127.0.98.62:33857)
0927 03:31:52.014107 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:52.014109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:52.014373 (+ 264us) server_negotiation.cc:407] Connection header received
0927 03:31:52.015063 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:52.015064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:52.015077 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:52.015132 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:52.015848 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:52.016355 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:52.017352 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:52.017562 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:52.017641 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:52.018544 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:52.018546 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:52.018546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:31:52.018551 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:52.018567 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:31:52.018571 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:52.018619 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:52.018712 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 03:31:52.018740 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:31:52.019040 18407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:52.014065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33857 (local address 127.0.98.3:52627)
0927 03:31:52.014261 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:52.014306 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:52.014958 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:52.015206 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:52.015211 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:52.015220 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:31:52.015747 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:52.015751 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:52.016462 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:52.016463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:52.017250 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:52.017252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:52.017648 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:52.017649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:52.017788 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:52.018380 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:31:52.018381 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:31:52.018410 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:31:52.018432 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:31:52.018618 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:31:52.018623 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:31:52.018625 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:31:52.018716 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 03:31:52.018816 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260927 03:31:52.019544 18405 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33857
I20260927 03:31:52.019665 18405 heartbeater.cc:464] Registering TS with master...
I20260927 03:31:52.019837 18405 heartbeater.cc:511] Master 127.0.98.62:33857 requested a full tablet report, sending...
I20260927 03:31:52.020175 17954 ts_manager.cc:194] Registered new tserver with Master: d1f73f0025ed44c4869a6e70ae564984 (127.0.98.3:36131)
I20260927 03:31:52.020551 17954 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:52627
I20260927 03:31:52.025594 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:31:52.029873 17995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:52.026991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58642 (local address 127.0.98.62:33857)
0927 03:31:52.027051 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:52.027053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:52.027136 (+ 83us) server_negotiation.cc:407] Connection header received
0927 03:31:52.027198 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:52.027198 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:52.027206 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:52.027253 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:52.027508 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:52.028018 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:52.028767 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:52.028950 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:52.029006 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:52.029681 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:52.029682 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:52.029682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:31:52.029686 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:52.029698 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:31:52.029701 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:52.029743 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:52.029798 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 03:31:52.029816 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 03:31:52.031602 17954 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:31:52.033129 17954 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:31:52.039692 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:52.035256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40691 (local address 127.0.0.1:58836)
0927 03:31:52.035804 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:52.035814 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:52.035892 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:52.036229 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:52.036229 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:52.036231 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:52.036267 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:52.036268 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:52.037097 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:52.037099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:52.038535 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:52.038537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:52.039400 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:52.039401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:52.039501 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:52.039506 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:31:52.039559 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 03:31:52.039603 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":23,"thread_start_us":494,"threads_started":1}
I20260927 03:31:52.039703 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:52.035542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58836 (local address 127.0.98.2:40691)
0927 03:31:52.035700 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:52.035705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:52.035867 (+ 162us) server_negotiation.cc:407] Connection header received
0927 03:31:52.036050 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:52.036054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:52.036073 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:52.036132 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:52.036337 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:52.037014 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:52.038625 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:52.039322 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:52.039394 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:52.039523 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:52.039582 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 03:31:52.039624 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
I20260927 03:31:52.039891 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:52.035253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36131 (local address 127.0.0.1:36748)
0927 03:31:52.035547 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:52.035596 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:52.035689 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:52.035980 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:52.035983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:52.035986 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:52.036055 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:52.036057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:52.037379 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:52.037381 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:52.038332 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:52.038335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:52.039343 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:52.039344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:52.039446 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:52.039463 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 03:31:52.039528 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 03:31:52.039586 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":76,"threads_started":1}
I20260927 03:31:52.039911 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:52.035337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36748 (local address 127.0.98.3:36131)
0927 03:31:52.035512 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:52.035518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:52.035775 (+ 257us) server_negotiation.cc:407] Connection header received
0927 03:31:52.035821 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:52.035823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:52.035838 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:52.035905 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:52.036134 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:52.037280 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:52.038417 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:52.039136 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:52.039211 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:52.039304 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:52.039649 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 03:31:52.039689 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":119,"threads_started":1}
I20260927 03:31:52.040633 18418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:52.035256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41647 (local address 127.0.0.1:55352)
0927 03:31:52.035786 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:52.035798 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:52.035863 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:52.036183 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:52.036183 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:52.036185 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:52.036229 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:52.036231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:52.037454 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:52.037455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:52.038533 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:52.038535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:52.040386 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:52.040387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:52.040454 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:52.040458 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:31:52.040529 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 03:31:52.040567 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":2,"thread_start_us":68,"threads_started":1}
I20260927 03:31:52.040660 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:52.035512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55352 (local address 127.0.98.1:41647)
0927 03:31:52.035799 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:52.035806 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:52.035851 (+ 45us) server_negotiation.cc:407] Connection header received
0927 03:31:52.036021 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:52.036023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:52.036038 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:52.036112 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:52.036331 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:52.037371 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:52.039499 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:52.040302 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:52.040375 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:52.040473 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:52.040543 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 03:31:52.040579 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":92,"threads_started":1}
I20260927 03:31:52.040668 18208 tablet_service.cc:1467] Processing CreateTablet for tablet e8bab21bf7b145af885b9521aac83410 (DEFAULT_TABLE table=test-table [id=05eb55beb1e3404c92ef61e03ac39240]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:31:52.040849 18339 tablet_service.cc:1467] Processing CreateTablet for tablet e8bab21bf7b145af885b9521aac83410 (DEFAULT_TABLE table=test-table [id=05eb55beb1e3404c92ef61e03ac39240]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:31:52.041105 18208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8bab21bf7b145af885b9521aac83410. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:31:52.041230 18339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8bab21bf7b145af885b9521aac83410. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:31:52.044096 18077 tablet_service.cc:1467] Processing CreateTablet for tablet e8bab21bf7b145af885b9521aac83410 (DEFAULT_TABLE table=test-table [id=05eb55beb1e3404c92ef61e03ac39240]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:31:52.044422 18077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8bab21bf7b145af885b9521aac83410. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:31:52.047036 18420 tablet_bootstrap.cc:492] T e8bab21bf7b145af885b9521aac83410 P 049ba44da45346b788359a510ea6549b: Bootstrap starting.
I20260927 03:31:52.047184 18421 tablet_bootstrap.cc:492] T e8bab21bf7b145af885b9521aac83410 P d1f73f0025ed44c4869a6e70ae564984: Bootstrap starting.
I20260927 03:31:52.047387 18419 tablet_bootstrap.cc:492] T e8bab21bf7b145af885b9521aac83410 P fa9cc2049c1b437ba32625e979cbd0a1: Bootstrap starting.
I20260927 03:31:52.047747 18420 tablet_bootstrap.cc:654] T e8bab21bf7b145af885b9521aac83410 P 049ba44da45346b788359a510ea6549b: Neither blocks nor log segments found. Creating new log.
I20260927 03:31:52.047896 18421 tablet_bootstrap.cc:654] T e8bab21bf7b145af885b9521aac83410 P d1f73f0025ed44c4869a6e70ae564984: Neither blocks nor log segments found. Creating new log.
I20260927 03:31:52.048030 18419 tablet_bootstrap.cc:654] T e8bab21bf7b145af885b9521aac83410 P fa9cc2049c1b437ba32625e979cbd0a1: Neither blocks nor log segments found. Creating new log.
I20260927 03:31:52.048105 18420 log.cc:824] T e8bab21bf7b145af885b9521aac83410 P 049ba44da45346b788359a510ea6549b: Log is configured to *not* fsync() on all Append() calls
I20260927 03:31:52.048233 18421 log.cc:824] T e8bab21bf7b145af885b9521aac83410 P d1f73f0025ed44c4869a6e70ae564984: Log is configured to *not* fsync() on all Append() calls
I20260927 03:31:52.048333 18419 log.cc:824] T e8bab21bf7b145af885b9521aac83410 P fa9cc2049c1b437ba32625e979cbd0a1: Log is configured to *not* fsync() on all Append() calls
I20260927 03:31:52.048905 18421 tablet_bootstrap.cc:492] T e8bab21bf7b145af885b9521aac83410 P d1f73f0025ed44c4869a6e70ae564984: No bootstrap required, opened a new log
I20260927 03:31:52.048919 18420 tablet_bootstrap.cc:492] T e8bab21bf7b145af885b9521aac83410 P 049ba44da45346b788359a510ea6549b: No bootstrap required, opened a new log
I20260927 03:31:52.048986 18420 ts_tablet_manager.cc:1397] T e8bab21bf7b145af885b9521aac83410 P 049ba44da45346b788359a510ea6549b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:31:52.048986 18421 ts_tablet_manager.cc:1397] T e8bab21bf7b145af885b9521aac83410 P d1f73f0025ed44c4869a6e70ae564984: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:31:52.049067 18419 tablet_bootstrap.cc:492] T e8bab21bf7b145af885b9521aac83410 P fa9cc2049c1b437ba32625e979cbd0a1: No bootstrap required, opened a new log
I20260927 03:31:52.049134 18419 ts_tablet_manager.cc:1397] T e8bab21bf7b145af885b9521aac83410 P fa9cc2049c1b437ba32625e979cbd0a1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:31:52.050982 18421 raft_consensus.cc:348] T e8bab21bf7b145af885b9521aac83410 P d1f73f0025ed44c4869a6e70ae564984 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f73f0025ed44c4869a6e70ae564984" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36131 } } peers { permanent_uuid: "fa9cc2049c1b437ba32625e979cbd0a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40691 } } peers { permanent_uuid: "049ba44da45346b788359a510ea6549b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41647 } }
I20260927 03:31:52.051133 18421 raft_consensus.cc:374] T e8bab21bf7b145af885b9521aac83410 P d1f73f0025ed44c4869a6e70ae564984 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:31:52.051177 18421 raft_consensus.cc:729] T e8bab21bf7b145af885b9521aac83410 P d1f73f0025ed44c4869a6e70ae564984 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f73f0025ed44c4869a6e70ae564984, State: Initialized, Role: FOLLOWER
I20260927 03:31:52.051285 18421 consensus_queue.cc:260] T e8bab21bf7b145af885b9521aac83410 P d1f73f0025ed44c4869a6e70ae564984 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f73f0025ed44c4869a6e70ae564984" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36131 } } peers { permanent_uuid: "fa9cc2049c1b437ba32625e979cbd0a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40691 } } peers { permanent_uuid: "049ba44da45346b788359a510ea6549b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41647 } }
I20260927 03:31:52.051568 18421 ts_tablet_manager.cc:1428] T e8bab21bf7b145af885b9521aac83410 P d1f73f0025ed44c4869a6e70ae564984: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 03:31:52.051702 18405 heartbeater.cc:503] Master 127.0.98.62:33857 was elected leader, sending a full tablet report...
I20260927 03:31:52.051615 18420 raft_consensus.cc:348] T e8bab21bf7b145af885b9521aac83410 P 049ba44da45346b788359a510ea6549b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f73f0025ed44c4869a6e70ae564984" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36131 } } peers { permanent_uuid: "fa9cc2049c1b437ba32625e979cbd0a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40691 } } peers { permanent_uuid: "049ba44da45346b788359a510ea6549b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41647 } }
I20260927 03:31:52.051765 18420 raft_consensus.cc:374] T e8bab21bf7b145af885b9521aac83410 P 049ba44da45346b788359a510ea6549b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:31:52.051800 18420 raft_consensus.cc:729] T e8bab21bf7b145af885b9521aac83410 P 049ba44da45346b788359a510ea6549b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 049ba44da45346b788359a510ea6549b, State: Initialized, Role: FOLLOWER
I20260927 03:31:52.051911 18420 consensus_queue.cc:260] T e8bab21bf7b145af885b9521aac83410 P 049ba44da45346b788359a510ea6549b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f73f0025ed44c4869a6e70ae564984" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36131 } } peers { permanent_uuid: "fa9cc2049c1b437ba32625e979cbd0a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40691 } } peers { permanent_uuid: "049ba44da45346b788359a510ea6549b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41647 } }
I20260927 03:31:52.052177 18419 raft_consensus.cc:348] T e8bab21bf7b145af885b9521aac83410 P fa9cc2049c1b437ba32625e979cbd0a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f73f0025ed44c4869a6e70ae564984" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36131 } } peers { permanent_uuid: "fa9cc2049c1b437ba32625e979cbd0a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40691 } } peers { permanent_uuid: "049ba44da45346b788359a510ea6549b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41647 } }
I20260927 03:31:52.052320 18419 raft_consensus.cc:374] T e8bab21bf7b145af885b9521aac83410 P fa9cc2049c1b437ba32625e979cbd0a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:31:52.052354 18419 raft_consensus.cc:729] T e8bab21bf7b145af885b9521aac83410 P fa9cc2049c1b437ba32625e979cbd0a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa9cc2049c1b437ba32625e979cbd0a1, State: Initialized, Role: FOLLOWER
I20260927 03:31:52.052470 18419 consensus_queue.cc:260] T e8bab21bf7b145af885b9521aac83410 P fa9cc2049c1b437ba32625e979cbd0a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f73f0025ed44c4869a6e70ae564984" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36131 } } peers { permanent_uuid: "fa9cc2049c1b437ba32625e979cbd0a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40691 } } peers { permanent_uuid: "049ba44da45346b788359a510ea6549b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41647 } }
I20260927 03:31:52.052665 18420 ts_tablet_manager.cc:1428] T e8bab21bf7b145af885b9521aac83410 P 049ba44da45346b788359a510ea6549b: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 03:31:52.052711 18419 ts_tablet_manager.cc:1428] T e8bab21bf7b145af885b9521aac83410 P fa9cc2049c1b437ba32625e979cbd0a1: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260927 03:31:52.052778 18274 heartbeater.cc:503] Master 127.0.98.62:33857 was elected leader, sending a full tablet report...
I20260927 03:31:52.053054 18143 heartbeater.cc:503] Master 127.0.98.62:33857 was elected leader, sending a full tablet report...
I20260927 03:31:52.116330 18426 raft_consensus.cc:482] T e8bab21bf7b145af885b9521aac83410 P 049ba44da45346b788359a510ea6549b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:31:52.116462 18426 raft_consensus.cc:504] T e8bab21bf7b145af885b9521aac83410 P 049ba44da45346b788359a510ea6549b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f73f0025ed44c4869a6e70ae564984" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36131 } } peers { permanent_uuid: "fa9cc2049c1b437ba32625e979cbd0a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40691 } } peers { permanent_uuid: "049ba44da45346b788359a510ea6549b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41647 } }
I20260927 03:31:52.116776 18426 leader_election.cc:288] T e8bab21bf7b145af885b9521aac83410 P 049ba44da45346b788359a510ea6549b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f73f0025ed44c4869a6e70ae564984 (127.0.98.3:36131), fa9cc2049c1b437ba32625e979cbd0a1 (127.0.98.2:40691)
I20260927 03:31:52.120775 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:52.117074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47585 (local address 127.0.98.3:36131)
0927 03:31:52.117116 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:52.117118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:52.117245 (+ 127us) server_negotiation.cc:407] Connection header received
0927 03:31:52.117284 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:52.117285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:52.117295 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:52.117350 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:52.117506 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:52.118255 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:52.119852 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:52.120415 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:52.120494 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:52.120505 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:52.120661 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 03:31:52.120704 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:31:52.120820 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:52.116971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36131 (local address 127.0.98.1:47585)
0927 03:31:52.117171 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:52.117178 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:52.117219 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:52.117364 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:52.117364 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:52.117366 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:52.117437 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:52.117438 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:52.118352 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:52.118353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:52.119767 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:52.119768 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:52.120509 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:52.120510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:52.120581 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:52.120585 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:31:52.120653 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 03:31:52.120696 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":97,"thread_start_us":118,"threads_started":1}
I20260927 03:31:52.120837 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:52.117060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43785 (local address 127.0.98.2:40691)
0927 03:31:52.117107 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:52.117109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:52.117147 (+ 38us) server_negotiation.cc:407] Connection header received
0927 03:31:52.117181 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:52.117181 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:52.117190 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:52.117263 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:52.117513 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:52.118252 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:52.119852 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:52.120440 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:52.120531 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:52.120540 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:52.120727 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 03:31:52.120771 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:31:52.120839 18145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:52.116971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40691 (local address 127.0.98.1:43785)
0927 03:31:52.117029 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:52.117042 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:52.117089 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:52.117289 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:52.117289 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:52.117291 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:52.117395 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:52.117398 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:52.118352 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:52.118353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:52.119761 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:52.119764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:52.120558 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:52.120559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:52.120633 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:52.120636 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:31:52.120706 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 03:31:52.120752 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":93}
I20260927 03:31:52.121192 18359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8bab21bf7b145af885b9521aac83410" candidate_uuid: "049ba44da45346b788359a510ea6549b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f73f0025ed44c4869a6e70ae564984" is_pre_election: true
I20260927 03:31:52.121213 18228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8bab21bf7b145af885b9521aac83410" candidate_uuid: "049ba44da45346b788359a510ea6549b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa9cc2049c1b437ba32625e979cbd0a1" is_pre_election: true
I20260927 03:31:52.121354 18359 raft_consensus.cc:2454] T e8bab21bf7b145af885b9521aac83410 P d1f73f0025ed44c4869a6e70ae564984 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 049ba44da45346b788359a510ea6549b in term 0.
I20260927 03:31:52.121367 18228 raft_consensus.cc:2454] T e8bab21bf7b145af885b9521aac83410 P fa9cc2049c1b437ba32625e979cbd0a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 049ba44da45346b788359a510ea6549b in term 0.
I20260927 03:31:52.121585 18033 leader_election.cc:302] T e8bab21bf7b145af885b9521aac83410 P 049ba44da45346b788359a510ea6549b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 049ba44da45346b788359a510ea6549b, d1f73f0025ed44c4869a6e70ae564984; no voters:
I20260927 03:31:52.121737 18426 raft_consensus.cc:2781] T e8bab21bf7b145af885b9521aac83410 P 049ba44da45346b788359a510ea6549b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:31:52.121803 18426 raft_consensus.cc:482] T e8bab21bf7b145af885b9521aac83410 P 049ba44da45346b788359a510ea6549b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:31:52.121821 18426 raft_consensus.cc:3037] T e8bab21bf7b145af885b9521aac83410 P 049ba44da45346b788359a510ea6549b [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:31:52.122589 18426 raft_consensus.cc:504] T e8bab21bf7b145af885b9521aac83410 P 049ba44da45346b788359a510ea6549b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f73f0025ed44c4869a6e70ae564984" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36131 } } peers { permanent_uuid: "fa9cc2049c1b437ba32625e979cbd0a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40691 } } peers { permanent_uuid: "049ba44da45346b788359a510ea6549b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41647 } }
I20260927 03:31:52.122736 18426 leader_election.cc:288] T e8bab21bf7b145af885b9521aac83410 P 049ba44da45346b788359a510ea6549b [CANDIDATE]: Term 1 election: Requested vote from peers d1f73f0025ed44c4869a6e70ae564984 (127.0.98.3:36131), fa9cc2049c1b437ba32625e979cbd0a1 (127.0.98.2:40691)
I20260927 03:31:52.122891 18228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8bab21bf7b145af885b9521aac83410" candidate_uuid: "049ba44da45346b788359a510ea6549b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa9cc2049c1b437ba32625e979cbd0a1"
I20260927 03:31:52.122982 18359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8bab21bf7b145af885b9521aac83410" candidate_uuid: "049ba44da45346b788359a510ea6549b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f73f0025ed44c4869a6e70ae564984"
I20260927 03:31:52.122973 18228 raft_consensus.cc:3037] T e8bab21bf7b145af885b9521aac83410 P fa9cc2049c1b437ba32625e979cbd0a1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:31:52.123068 18359 raft_consensus.cc:3037] T e8bab21bf7b145af885b9521aac83410 P d1f73f0025ed44c4869a6e70ae564984 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:31:52.123788 18228 raft_consensus.cc:2454] T e8bab21bf7b145af885b9521aac83410 P fa9cc2049c1b437ba32625e979cbd0a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 049ba44da45346b788359a510ea6549b in term 1.
I20260927 03:31:52.123904 18359 raft_consensus.cc:2454] T e8bab21bf7b145af885b9521aac83410 P d1f73f0025ed44c4869a6e70ae564984 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 049ba44da45346b788359a510ea6549b in term 1.
I20260927 03:31:52.123957 18032 leader_election.cc:302] T e8bab21bf7b145af885b9521aac83410 P 049ba44da45346b788359a510ea6549b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 049ba44da45346b788359a510ea6549b, fa9cc2049c1b437ba32625e979cbd0a1; no voters:
I20260927 03:31:52.124047 18426 raft_consensus.cc:2781] T e8bab21bf7b145af885b9521aac83410 P 049ba44da45346b788359a510ea6549b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:31:52.124212 18426 raft_consensus.cc:686] T e8bab21bf7b145af885b9521aac83410 P 049ba44da45346b788359a510ea6549b [term 1 LEADER]: Becoming Leader. State: Replica: 049ba44da45346b788359a510ea6549b, State: Running, Role: LEADER
W20260927 03:31:52.124279 18275 tablet.cc:2369] T e8bab21bf7b145af885b9521aac83410 P fa9cc2049c1b437ba32625e979cbd0a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:31:52.124317 18426 consensus_queue.cc:237] T e8bab21bf7b145af885b9521aac83410 P 049ba44da45346b788359a510ea6549b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f73f0025ed44c4869a6e70ae564984" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36131 } } peers { permanent_uuid: "fa9cc2049c1b437ba32625e979cbd0a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40691 } } peers { permanent_uuid: "049ba44da45346b788359a510ea6549b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41647 } }
I20260927 03:31:52.125164 17954 catalog_manager.cc:5495] T e8bab21bf7b145af885b9521aac83410 P 049ba44da45346b788359a510ea6549b reported cstate change: term changed from 0 to 1, leader changed from <none> to 049ba44da45346b788359a510ea6549b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "049ba44da45346b788359a510ea6549b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f73f0025ed44c4869a6e70ae564984" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa9cc2049c1b437ba32625e979cbd0a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "049ba44da45346b788359a510ea6549b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41647 } health_report { overall_health: HEALTHY } } }
I20260927 03:31:52.133728 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:52.131481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55356 (local address 127.0.98.1:41647)
0927 03:31:52.131531 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:52.131533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:52.131543 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:31:52.131582 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:52.131582 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:52.131608 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:52.131675 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:31:52.131773 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:52.132240 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:52.132962 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:52.133159 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:52.133227 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:52.133353 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:31:52.133465 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:31:52.133526 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:52.133557 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 03:31:52.133575 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:31:52.135573 18359 raft_consensus.cc:1261] T e8bab21bf7b145af885b9521aac83410 P d1f73f0025ed44c4869a6e70ae564984 [term 1 FOLLOWER]: Refusing update from remote peer 049ba44da45346b788359a510ea6549b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:31:52.135573 18228 raft_consensus.cc:1261] T e8bab21bf7b145af885b9521aac83410 P fa9cc2049c1b437ba32625e979cbd0a1 [term 1 FOLLOWER]: Refusing update from remote peer 049ba44da45346b788359a510ea6549b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:31:52.135946 18426 consensus_queue.cc:1035] T e8bab21bf7b145af885b9521aac83410 P 049ba44da45346b788359a510ea6549b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f73f0025ed44c4869a6e70ae564984" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:31:52.136099 18429 consensus_queue.cc:1035] T e8bab21bf7b145af885b9521aac83410 P 049ba44da45346b788359a510ea6549b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa9cc2049c1b437ba32625e979cbd0a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:31:52.138418 18434 mvcc.cc:205] Tried to move back new op lower bound from 7333805720104509440 to 7333805720061894656. Current Snapshot: MvccSnapshot[applied={T|T < 7333805720104509440}]
I20260927 03:31:52.138983 18433 mvcc.cc:205] Tried to move back new op lower bound from 7333805720104509440 to 7333805720061894656. Current Snapshot: MvccSnapshot[applied={T|T < 7333805720104509440}]
I20260927 03:31:52.141110 17954 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:31:52.141193 17954 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:31:52.141902 17954 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e7dfef021d24f0dbed7379047dc77df: Sending DeleteTablet for 3 replicas of tablet e8bab21bf7b145af885b9521aac83410
I20260927 03:31:52.142241 18339 tablet_service.cc:1514] Processing DeleteTablet for tablet e8bab21bf7b145af885b9521aac83410 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:31:52 UTC) from {username='slave'} at 127.0.0.1:36748
I20260927 03:31:52.142287 18208 tablet_service.cc:1514] Processing DeleteTablet for tablet e8bab21bf7b145af885b9521aac83410 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:31:52 UTC) from {username='slave'} at 127.0.0.1:58836
I20260927 03:31:52.142515 18077 tablet_service.cc:1514] Processing DeleteTablet for tablet e8bab21bf7b145af885b9521aac83410 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:31:52 UTC) from {username='slave'} at 127.0.0.1:55352
I20260927 03:31:52.142750 18441 tablet_replica.cc:330] T e8bab21bf7b145af885b9521aac83410 P d1f73f0025ed44c4869a6e70ae564984: stopping tablet replica
I20260927 03:31:52.142762 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 18015
I20260927 03:31:52.142854 18441 raft_consensus.cc:2229] T e8bab21bf7b145af885b9521aac83410 P d1f73f0025ed44c4869a6e70ae564984 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:31:52.142956 18441 raft_consensus.cc:2258] T e8bab21bf7b145af885b9521aac83410 P d1f73f0025ed44c4869a6e70ae564984 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:31:52.143801 18441 ts_tablet_manager.cc:1905] T e8bab21bf7b145af885b9521aac83410 P d1f73f0025ed44c4869a6e70ae564984: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:31:52.145787 18441 ts_tablet_manager.cc:1918] T e8bab21bf7b145af885b9521aac83410 P d1f73f0025ed44c4869a6e70ae564984: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:31:52.145846 18441 log.cc:1196] T e8bab21bf7b145af885b9521aac83410 P d1f73f0025ed44c4869a6e70ae564984: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790479675865103-392-0/minicluster-data/ts-2/wal/wals/e8bab21bf7b145af885b9521aac83410
I20260927 03:31:52.146113 18441 ts_tablet_manager.cc:1939] T e8bab21bf7b145af885b9521aac83410 P d1f73f0025ed44c4869a6e70ae564984: Deleting consensus metadata
I20260927 03:31:52.146512 17942 catalog_manager.cc:4841] TS d1f73f0025ed44c4869a6e70ae564984 (127.0.98.3:36131): tablet e8bab21bf7b145af885b9521aac83410 (table test-table [id=05eb55beb1e3404c92ef61e03ac39240]) successfully deleted
I20260927 03:31:52.148057 18442 tablet_replica.cc:330] T e8bab21bf7b145af885b9521aac83410 P fa9cc2049c1b437ba32625e979cbd0a1: stopping tablet replica
I20260927 03:31:52.148198 18442 raft_consensus.cc:2229] T e8bab21bf7b145af885b9521aac83410 P fa9cc2049c1b437ba32625e979cbd0a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:31:52.148324 18442 raft_consensus.cc:2258] T e8bab21bf7b145af885b9521aac83410 P fa9cc2049c1b437ba32625e979cbd0a1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 03:31:52.148828 17940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41647 (error 108)
I20260927 03:31:52.148895 18442 ts_tablet_manager.cc:1905] T e8bab21bf7b145af885b9521aac83410 P fa9cc2049c1b437ba32625e979cbd0a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:31:52.150240 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 18146
I20260927 03:31:52.152664 18418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:52.152525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41647 (local address 127.0.0.1:55364)
0927 03:31:52.152585 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:52.152632 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 03:31:52.152838 17940 catalog_manager.cc:4584] TS 049ba44da45346b788359a510ea6549b (127.0.98.1:41647): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e8bab21bf7b145af885b9521aac83410: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41647: connect: Connection refused (error 111)
I20260927 03:31:52.155493 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 18277
I20260927 03:31:52.155514 18418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:52.155370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40691 (local address 127.0.0.1:58844)
0927 03:31:52.155420 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:52.155491 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 03:31:52.160167 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 17924
2026-09-27T03:31:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (782 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:31:52 dist-test-slave-j0hf krb5kdc[18447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:31:52 dist-test-slave-j0hf krb5kdc[18447](info): set up 2 sockets
Sep 27 03:31:52 dist-test-slave-j0hf krb5kdc[18447](info): commencing operation
krb5kdc: starting...
W20260927 03:31:53.199271 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:31:53 dist-test-slave-j0hf krb5kdc[18447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:31:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:31:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:31:53.318768 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43279
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:31:53.400635 18460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:53.400897 18460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:53.400940 18460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:53.402211 18460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:31:53.402280 18460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:53.402305 18460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:53.402328 18460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:31:53.402355 18460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:31:53.403891 18460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:53.404196 18460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:53.404438 18460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:31:53.409338 18466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:53.409348 18465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:53.409386 18468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:53.409571 18460 server_base.cc:1034] running on GCE node
I20260927 03:31:53.409834 18460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:53.410125 18460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:53.411285 18460 hybrid_clock.cc:648] HybridClock initialized: now 1790479913411288 us; error 43 us; skew 500 ppm
Sep 27 03:31:53 dist-test-slave-j0hf krb5kdc[18447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479913, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:31:53.413694 18460 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:31:53.414134 18460 webserver.cc:460] Webserver started at http://127.0.98.62:34139/ using document root <none> and password file <none>
I20260927 03:31:53.414376 18460 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:53.414453 18460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:53.414615 18460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:31:53.415760 18460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "a3b8fac51c4d4dfca5c2e755a47d75db"
format_stamp: "Formatted at 2026-09-27 03:31:53 on dist-test-slave-j0hf"
server_key: "cbc71694f4115b9f0508aa11740a5db0"
server_key_iv: "73c9e39125d2f1d6cbbfaa72970ba969"
server_key_version: "encryptionkey@0"
I20260927 03:31:53.416128 18460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "a3b8fac51c4d4dfca5c2e755a47d75db"
format_stamp: "Formatted at 2026-09-27 03:31:53 on dist-test-slave-j0hf"
server_key: "cbc71694f4115b9f0508aa11740a5db0"
server_key_iv: "73c9e39125d2f1d6cbbfaa72970ba969"
server_key_version: "encryptionkey@0"
I20260927 03:31:53.417483 18460 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:31:53.418186 18475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:53.418359 18460 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:31:53.418470 18460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "a3b8fac51c4d4dfca5c2e755a47d75db"
format_stamp: "Formatted at 2026-09-27 03:31:53 on dist-test-slave-j0hf"
server_key: "cbc71694f4115b9f0508aa11740a5db0"
server_key_iv: "73c9e39125d2f1d6cbbfaa72970ba969"
server_key_version: "encryptionkey@0"
I20260927 03:31:53.418546 18460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:53.439154 18460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:53.441056 18460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:53.441237 18460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:53.445147 18460 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43279
I20260927 03:31:53.445186 18527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43279 every 8 connection(s)
I20260927 03:31:53.445576 18460 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:31:53.446471 18528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:31:53.448763 18528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3b8fac51c4d4dfca5c2e755a47d75db: Bootstrap starting.
I20260927 03:31:53.449361 18528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3b8fac51c4d4dfca5c2e755a47d75db: Neither blocks nor log segments found. Creating new log.
I20260927 03:31:53.449659 18528 log.cc:824] T 00000000000000000000000000000000 P a3b8fac51c4d4dfca5c2e755a47d75db: Log is configured to *not* fsync() on all Append() calls
I20260927 03:31:53.450369 18528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3b8fac51c4d4dfca5c2e755a47d75db: No bootstrap required, opened a new log
I20260927 03:31:53.452042 18528 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3b8fac51c4d4dfca5c2e755a47d75db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b8fac51c4d4dfca5c2e755a47d75db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43279 } }
I20260927 03:31:53.452180 18528 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3b8fac51c4d4dfca5c2e755a47d75db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:31:53.452203 18528 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3b8fac51c4d4dfca5c2e755a47d75db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3b8fac51c4d4dfca5c2e755a47d75db, State: Initialized, Role: FOLLOWER
I20260927 03:31:53.452395 18528 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3b8fac51c4d4dfca5c2e755a47d75db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b8fac51c4d4dfca5c2e755a47d75db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43279 } }
I20260927 03:31:53.452481 18528 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3b8fac51c4d4dfca5c2e755a47d75db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:31:53.452531 18528 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3b8fac51c4d4dfca5c2e755a47d75db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:31:53.452581 18528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3b8fac51c4d4dfca5c2e755a47d75db [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:31:53.453105 18528 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3b8fac51c4d4dfca5c2e755a47d75db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b8fac51c4d4dfca5c2e755a47d75db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43279 } }
I20260927 03:31:53.453217 18528 leader_election.cc:302] T 00000000000000000000000000000000 P a3b8fac51c4d4dfca5c2e755a47d75db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3b8fac51c4d4dfca5c2e755a47d75db; no voters:
I20260927 03:31:53.453423 18528 leader_election.cc:288] T 00000000000000000000000000000000 P a3b8fac51c4d4dfca5c2e755a47d75db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:31:53.453462 18531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3b8fac51c4d4dfca5c2e755a47d75db [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:31:53.453644 18531 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3b8fac51c4d4dfca5c2e755a47d75db [term 1 LEADER]: Becoming Leader. State: Replica: a3b8fac51c4d4dfca5c2e755a47d75db, State: Running, Role: LEADER
I20260927 03:31:53.453685 18528 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3b8fac51c4d4dfca5c2e755a47d75db [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:31:53.453730 18531 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3b8fac51c4d4dfca5c2e755a47d75db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b8fac51c4d4dfca5c2e755a47d75db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43279 } }
I20260927 03:31:53.453992 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 18460
I20260927 03:31:53.454131 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:31:53.454371 392 external_mini_cluster.cc:1428] Setting key e1ed3cbede3b71b52f22803b5e20779a
I20260927 03:31:53.454965 18532 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3b8fac51c4d4dfca5c2e755a47d75db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3b8fac51c4d4dfca5c2e755a47d75db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b8fac51c4d4dfca5c2e755a47d75db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43279 } } }
I20260927 03:31:53.455101 18532 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3b8fac51c4d4dfca5c2e755a47d75db [sys.catalog]: This master's current role is: LEADER
I20260927 03:31:53.455231 18533 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3b8fac51c4d4dfca5c2e755a47d75db [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3b8fac51c4d4dfca5c2e755a47d75db. Latest consensus state: current_term: 1 leader_uuid: "a3b8fac51c4d4dfca5c2e755a47d75db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3b8fac51c4d4dfca5c2e755a47d75db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43279 } } }
I20260927 03:31:53.455324 18533 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3b8fac51c4d4dfca5c2e755a47d75db [sys.catalog]: This master's current role is: LEADER
W20260927 03:31:53.455816 18548 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a3b8fac51c4d4dfca5c2e755a47d75db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:31:53.455878 18548 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:31:53.455976 18546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:31:53.456202 18546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:31:53.457844 18546 catalog_manager.cc:1347] Generated new cluster ID: 460126fca93545b6863d21d903f9f76f
I20260927 03:31:53.457896 18546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:31:53 dist-test-slave-j0hf krb5kdc[18447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:31:53.464990 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:53.455032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57730 (local address 127.0.98.62:43279)
0927 03:31:53.455681 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:53.455691 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:53.455728 (+ 37us) server_negotiation.cc:407] Connection header received
0927 03:31:53.455848 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:53.455855 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:53.455913 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:53.456047 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:53.456883 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:53.458103 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:53.458860 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:53.459308 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:53.459426 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:53.461756 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:53.461760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:53.461760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:31:53.461771 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:53.463423 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:53.463847 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:53.463850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:53.463850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:53.463944 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:53.464156 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:53.464156 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:53.464157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:53.464386 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:53.464593 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:53.464659 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 03:31:53.464794 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":88,"threads_started":1}
I20260927 03:31:53.512534 18546 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:31:53.513103 18546 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:31:53.516142 18546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3b8fac51c4d4dfca5c2e755a47d75db: Generated new TSK 0
I20260927 03:31:53.516333 18546 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:31:53.545855 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43279
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:31:53.648617 18554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:53.648885 18554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:53.648927 18554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:53.650135 18554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:53.650213 18554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:53.650257 18554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:31:53.651858 18554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:53.652139 18554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:53.652382 18554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:31:53.657653 18562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:53.657738 18559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:53.657649 18560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:53.658004 18554 server_base.cc:1034] running on GCE node
I20260927 03:31:53.658246 18554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:53.658501 18554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:53.659662 18554 hybrid_clock.cc:648] HybridClock initialized: now 1790479913659647 us; error 47 us; skew 500 ppm
Sep 27 03:31:53 dist-test-slave-j0hf krb5kdc[18447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479913, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:31:53.662281 18554 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:31:53.662765 18554 webserver.cc:460] Webserver started at http://127.0.98.1:45335/ using document root <none> and password file <none>
I20260927 03:31:53.662961 18554 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:53.663000 18554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:53.663089 18554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:31:53.664192 18554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-0/data/instance:
uuid: "eab5841c2f0b4704b85df87735743a94"
format_stamp: "Formatted at 2026-09-27 03:31:53 on dist-test-slave-j0hf"
server_key: "f27ab492a85540ca4e178fc602989a5a"
server_key_iv: "547f24a86deef4b4e0828f2ec9926287"
server_key_version: "encryptionkey@0"
I20260927 03:31:53.664503 18554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-0/wal/instance:
uuid: "eab5841c2f0b4704b85df87735743a94"
format_stamp: "Formatted at 2026-09-27 03:31:53 on dist-test-slave-j0hf"
server_key: "f27ab492a85540ca4e178fc602989a5a"
server_key_iv: "547f24a86deef4b4e0828f2ec9926287"
server_key_version: "encryptionkey@0"
I20260927 03:31:53.665772 18554 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:31:53.666440 18569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:53.666627 18554 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:31:53.666718 18554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "eab5841c2f0b4704b85df87735743a94"
format_stamp: "Formatted at 2026-09-27 03:31:53 on dist-test-slave-j0hf"
server_key: "f27ab492a85540ca4e178fc602989a5a"
server_key_iv: "547f24a86deef4b4e0828f2ec9926287"
server_key_version: "encryptionkey@0"
I20260927 03:31:53.666774 18554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:53.675359 18554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:53.677253 18554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:53.677362 18554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:53.677542 18554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:31:53.677866 18554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:31:53.677901 18554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:53.677922 18554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:31:53.677977 18554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:53.683519 18554 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33319
I20260927 03:31:53.683597 18682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33319 every 8 connection(s)
I20260927 03:31:53.683941 18554 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:31:53 dist-test-slave-j0hf krb5kdc[18447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479913, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:31:53.691665 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 18554
I20260927 03:31:53.691793 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:31:53.692070 392 external_mini_cluster.cc:1428] Setting key d8509eb8827f6ae0643da5ec28b2b070
I20260927 03:31:53.693585 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:53.684744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49619 (local address 127.0.98.62:43279)
0927 03:31:53.684792 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:53.684794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:53.685003 (+ 209us) server_negotiation.cc:407] Connection header received
0927 03:31:53.685901 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:53.685901 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:53.685914 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:53.685982 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:53.686577 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:53.687088 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:53.688167 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:53.688363 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:53.688414 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:53.690479 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:53.690480 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:53.690481 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:31:53.690491 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:53.692030 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:53.692517 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:53.692518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:53.692519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:53.692575 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:53.692907 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:53.692909 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:53.692909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:53.693036 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:53.693113 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:53.693409 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 03:31:53.693496 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:31:53.693862 18685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:53.684696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43279 (local address 127.0.98.1:49619)
0927 03:31:53.684856 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:53.684898 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:53.685810 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:53.686042 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:53.686045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:53.686305 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:31:53.686497 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:53.686501 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:53.687173 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:53.687174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:53.688087 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:53.688089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:53.688433 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:53.688434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:53.688534 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:53.689022 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:31:53.689031 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:31:53.690384 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:31:53.692114 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:31:53.692121 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:31:53.692123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:31:53.692426 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:31:53.692672 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:31:53.692672 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:31:53.692673 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:31:53.692820 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:31:53.693118 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:31:53.693125 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:31:53.693269 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 03:31:53.693375 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 03:31:53.693546 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260927 03:31:53.694437 18683 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43279
I20260927 03:31:53.694567 18683 heartbeater.cc:464] Registering TS with master...
I20260927 03:31:53.694773 18683 heartbeater.cc:511] Master 127.0.98.62:43279 requested a full tablet report, sending...
I20260927 03:31:53.695319 18492 ts_manager.cc:194] Registered new tserver with Master: eab5841c2f0b4704b85df87735743a94 (127.0.98.1:33319)
I20260927 03:31:53.695984 18492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49619
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:31:53.716270 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43279
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:31:53.799576 18688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:53.799793 18688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:53.799818 18688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:53.801261 18688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:53.801352 18688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:53.801442 18688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:31:53.803130 18688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:53.803385 18688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:53.803651 18688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:31:53.808693 18693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:53.808693 18694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:53.808779 18688 server_base.cc:1034] running on GCE node
W20260927 03:31:53.808693 18696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:53.809202 18688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:53.809461 18688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:53.810637 18688 hybrid_clock.cc:648] HybridClock initialized: now 1790479913810618 us; error 38 us; skew 500 ppm
Sep 27 03:31:53 dist-test-slave-j0hf krb5kdc[18447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479913, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:31:53.813186 18688 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:31:53.813673 18688 webserver.cc:460] Webserver started at http://127.0.98.2:33365/ using document root <none> and password file <none>
I20260927 03:31:53.813911 18688 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:53.813984 18688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:53.814127 18688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:31:53.815217 18688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-1/data/instance:
uuid: "6f640043b4ca4e4986886b6ac0cfd63f"
format_stamp: "Formatted at 2026-09-27 03:31:53 on dist-test-slave-j0hf"
server_key: "5b8dd77d610032e4875f0159e525eac1"
server_key_iv: "8c06d81505faa08760504274b959c22c"
server_key_version: "encryptionkey@0"
I20260927 03:31:53.815605 18688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6f640043b4ca4e4986886b6ac0cfd63f"
format_stamp: "Formatted at 2026-09-27 03:31:53 on dist-test-slave-j0hf"
server_key: "5b8dd77d610032e4875f0159e525eac1"
server_key_iv: "8c06d81505faa08760504274b959c22c"
server_key_version: "encryptionkey@0"
I20260927 03:31:53.816919 18688 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:31:53.817628 18703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:53.817801 18688 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:31:53.817891 18688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "6f640043b4ca4e4986886b6ac0cfd63f"
format_stamp: "Formatted at 2026-09-27 03:31:53 on dist-test-slave-j0hf"
server_key: "5b8dd77d610032e4875f0159e525eac1"
server_key_iv: "8c06d81505faa08760504274b959c22c"
server_key_version: "encryptionkey@0"
I20260927 03:31:53.817968 18688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:53.829418 18688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:53.832580 18688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:53.832741 18688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:53.833043 18688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:31:53.833456 18688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:31:53.833493 18688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:53.833539 18688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:31:53.833566 18688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:53.839097 18688 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33969
I20260927 03:31:53.839191 18816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33969 every 8 connection(s)
I20260927 03:31:53.839551 18688 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:31:53.841090 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 18688
I20260927 03:31:53.841183 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:31:53.841400 392 external_mini_cluster.cc:1428] Setting key 71a7fd574b2a18cead752b73cf0fc0eb
Sep 27 03:31:53 dist-test-slave-j0hf krb5kdc[18447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479913, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:31:53.850579 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:53.840295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51133 (local address 127.0.98.62:43279)
0927 03:31:53.840343 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:53.840346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:53.840558 (+ 212us) server_negotiation.cc:407] Connection header received
0927 03:31:53.841814 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:53.841815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:53.841831 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:53.841909 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:53.842724 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:53.843554 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:53.844605 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:53.844902 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:53.844972 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:53.847513 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:53.847514 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:53.847515 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:31:53.847524 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:53.849171 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:53.849580 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:53.849582 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:53.849582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:53.849646 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:53.849882 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:53.849884 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:53.849884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:53.850049 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:53.850125 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:53.850379 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 03:31:53.850471 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:31:53.850716 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:53.840258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43279 (local address 127.0.98.2:51133)
0927 03:31:53.840421 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:53.840473 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:53.841546 (+ 1073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:53.842012 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:53.842016 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:53.842406 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:31:53.842624 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:53.842630 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:53.843658 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:53.843659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:53.844511 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:53.844514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:53.845007 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:53.845008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:53.845167 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:53.845717 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:31:53.845729 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:31:53.847371 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:31:53.849266 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:31:53.849272 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:31:53.849274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:31:53.849500 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:31:53.849717 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:31:53.849718 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:31:53.849718 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:31:53.849814 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:31:53.850116 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:31:53.850120 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:31:53.850240 (+ 120us) client_negotiation.cc:769] Sending connection context
0927 03:31:53.850325 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 03:31:53.850450 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 03:31:53.851220 18817 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43279
I20260927 03:31:53.851310 18817 heartbeater.cc:464] Registering TS with master...
I20260927 03:31:53.851522 18817 heartbeater.cc:511] Master 127.0.98.62:43279 requested a full tablet report, sending...
I20260927 03:31:53.851944 18492 ts_manager.cc:194] Registered new tserver with Master: 6f640043b4ca4e4986886b6ac0cfd63f (127.0.98.2:33969)
I20260927 03:31:53.852337 18492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:31:53.867273 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43279
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:31:53.954679 18822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:53.954890 18822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:53.954916 18822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:53.956355 18822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:53.956408 18822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:53.956437 18822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:31:53.957981 18822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:53.958244 18822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:53.958487 18822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:31:53.963557 18827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:53.963562 18830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:53.963557 18828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:53.963791 18822 server_base.cc:1034] running on GCE node
I20260927 03:31:53.964000 18822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:53.964195 18822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:53.965366 18822 hybrid_clock.cc:648] HybridClock initialized: now 1790479913965348 us; error 29 us; skew 500 ppm
Sep 27 03:31:53 dist-test-slave-j0hf krb5kdc[18447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479913, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:31:53.967911 18822 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:31:53.968358 18822 webserver.cc:460] Webserver started at http://127.0.98.3:35117/ using document root <none> and password file <none>
I20260927 03:31:53.968595 18822 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:53.968683 18822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:53.968842 18822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:31:53.969866 18822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-2/data/instance:
uuid: "332386fac6d54adb835d5b7154f96675"
format_stamp: "Formatted at 2026-09-27 03:31:53 on dist-test-slave-j0hf"
server_key: "0e8505d42308cd81bdcf90508f296763"
server_key_iv: "c97f53d6d0dfdd173e043fe173b225b3"
server_key_version: "encryptionkey@0"
I20260927 03:31:53.970242 18822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-2/wal/instance:
uuid: "332386fac6d54adb835d5b7154f96675"
format_stamp: "Formatted at 2026-09-27 03:31:53 on dist-test-slave-j0hf"
server_key: "0e8505d42308cd81bdcf90508f296763"
server_key_iv: "c97f53d6d0dfdd173e043fe173b225b3"
server_key_version: "encryptionkey@0"
I20260927 03:31:53.971575 18822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:31:53.972335 18837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:53.972533 18822 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:53.972631 18822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "332386fac6d54adb835d5b7154f96675"
format_stamp: "Formatted at 2026-09-27 03:31:53 on dist-test-slave-j0hf"
server_key: "0e8505d42308cd81bdcf90508f296763"
server_key_iv: "c97f53d6d0dfdd173e043fe173b225b3"
server_key_version: "encryptionkey@0"
I20260927 03:31:53.972718 18822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:53.993830 18822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:53.996006 18822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:53.996186 18822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:53.996423 18822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:31:53.996742 18822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:31:53.996802 18822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:53.996843 18822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:31:53.996871 18822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:54.002730 18822 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35755
I20260927 03:31:54.002781 18950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35755 every 8 connection(s)
I20260927 03:31:54.003093 18822 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:31:54 dist-test-slave-j0hf krb5kdc[18447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479913, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:31:54.012578 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 18822
I20260927 03:31:54.012683 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:31:54.012893 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.003851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44355 (local address 127.0.98.62:43279)
0927 03:31:54.003932 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:54.003935 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:54.004046 (+ 111us) server_negotiation.cc:407] Connection header received
0927 03:31:54.005146 (+ 1100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:54.005147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:54.005160 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:54.005226 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:54.005905 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.006434 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.007510 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.007723 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.007821 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:54.010085 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:54.010087 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:54.010087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:31:54.010095 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:54.011411 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:54.011859 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:54.011861 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:54.011861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:54.011928 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:54.012177 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:54.012178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:54.012178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:54.012305 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:54.012384 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:54.012732 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 03:31:54.012826 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 03:31:54.012903 392 external_mini_cluster.cc:1428] Setting key 24af2ffe0922e7ab97e5ba7aa5034d49
I20260927 03:31:54.013252 18953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.003751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43279 (local address 127.0.98.3:44355)
0927 03:31:54.003908 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:54.003960 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:54.004921 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:54.005324 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:54.005328 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:54.005599 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:31:54.005803 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:54.005807 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.006530 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.006531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:54.007370 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:54.007372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.007830 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.007831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:54.007965 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:54.008531 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:31:54.008540 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:31:54.009974 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:31:54.011523 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:31:54.011530 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:31:54.011532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:31:54.011753 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:31:54.011994 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:31:54.011995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:31:54.011995 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:31:54.012106 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:31:54.012373 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:31:54.012379 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:31:54.012554 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 03:31:54.012685 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 03:31:54.012863 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
I20260927 03:31:54.013710 18951 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43279
I20260927 03:31:54.013827 18951 heartbeater.cc:464] Registering TS with master...
I20260927 03:31:54.014050 18951 heartbeater.cc:511] Master 127.0.98.62:43279 requested a full tablet report, sending...
I20260927 03:31:54.014437 18492 ts_manager.cc:194] Registered new tserver with Master: 332386fac6d54adb835d5b7154f96675 (127.0.98.3:35755)
I20260927 03:31:54.014859 18492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44355
I20260927 03:31:54.024827 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:31:54.032333 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.026124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57738 (local address 127.0.98.62:43279)
0927 03:31:54.026194 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:54.026196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:54.026296 (+ 100us) server_negotiation.cc:407] Connection header received
0927 03:31:54.026340 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:54.026341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:54.026349 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:54.026405 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:54.026953 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.027490 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.028177 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.028355 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.028403 (+ 48us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:54.029665 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:54.029666 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:54.029666 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:31:54.029673 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:54.031067 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:54.031551 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:54.031551 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:54.031552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:54.031603 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:54.031877 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:54.031878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:54.031878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:54.032006 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:54.032087 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:54.032222 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 03:31:54.032268 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 03:31:54.033944 18492 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:31:54.035570 18492 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:31:54.042006 18964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.037898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37896 (local address 127.0.98.3:35755)
0927 03:31:54.038314 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:54.038321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:54.038331 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:31:54.038370 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:54.038372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:54.038388 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:54.038458 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:54.038647 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.039513 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.040951 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.041574 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.041659 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:54.041721 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:54.041871 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 03:31:54.041912 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":99,"threads_started":1}
I20260927 03:31:54.041993 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.037826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35755 (local address 127.0.0.1:37896)
0927 03:31:54.038032 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:54.038075 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:54.038148 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:54.038480 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:54.038483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:54.038486 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:54.038564 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:54.038566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.039622 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.039623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:54.040861 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:54.040863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.041658 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.041659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:54.041738 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:54.041751 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 03:31:54.041853 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 03:31:54.041912 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":67,"threads_started":1}
I20260927 03:31:54.042232 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.038033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33969 (local address 127.0.0.1:60460)
0927 03:31:54.038233 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:54.038243 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:54.038280 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:54.038589 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:54.038590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:54.038593 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:54.038640 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:54.038642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.039982 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.039983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:54.041055 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:54.041057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.041985 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.041986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:54.042060 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:54.042065 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:31:54.042097 (+ 32us) client_negotiation.cc:240] Negotiation successful
0927 03:31:54.042138 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":25,"thread_start_us":67,"threads_started":1}
I20260927 03:31:54.042413 18965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.037971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38410 (local address 127.0.98.1:33319)
0927 03:31:54.038607 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:54.038612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:54.038622 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:31:54.038658 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:54.038660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:54.038677 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:54.038745 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:54.038954 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.040046 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.041266 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.041962 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.042020 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:54.042077 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:54.042273 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 03:31:54.042317 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":79,"threads_started":1}
I20260927 03:31:54.042815 18885 tablet_service.cc:1467] Processing CreateTablet for tablet cbc0471dc63b4a619c0b8d5542ba4e3d (DEFAULT_TABLE table=test-table [id=fe07448b611c402995fa57bd426e30b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:31:54.043102 18885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbc0471dc63b4a619c0b8d5542ba4e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:31:54.043696 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.038267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60460 (local address 127.0.98.2:33969)
0927 03:31:54.038402 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:54.038406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:54.038413 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:31:54.038444 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:54.038445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:54.038487 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:54.038712 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:54.038789 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.039877 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.041149 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.041879 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.043532 (+ 1653us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:54.043582 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:54.043604 (+ 22us) server_negotiation.cc:299] Negotiation successful
0927 03:31:54.043639 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260927 03:31:54.043879 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.037867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33319 (local address 127.0.0.1:38410)
0927 03:31:54.038138 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:54.038150 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:54.038201 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:54.038830 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:54.038831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:54.038833 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:54.038865 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:54.038867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.040150 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.040152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:54.041182 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:54.041185 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.042109 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.042110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:54.042158 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:54.042161 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:31:54.042205 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 03:31:54.042234 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":16,"thread_start_us":90,"threads_started":1}
I20260927 03:31:54.044385 18751 tablet_service.cc:1467] Processing CreateTablet for tablet cbc0471dc63b4a619c0b8d5542ba4e3d (DEFAULT_TABLE table=test-table [id=fe07448b611c402995fa57bd426e30b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:31:54.044726 18751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbc0471dc63b4a619c0b8d5542ba4e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:31:54.046310 18968 tablet_bootstrap.cc:492] T cbc0471dc63b4a619c0b8d5542ba4e3d P 332386fac6d54adb835d5b7154f96675: Bootstrap starting.
I20260927 03:31:54.047094 18968 tablet_bootstrap.cc:654] T cbc0471dc63b4a619c0b8d5542ba4e3d P 332386fac6d54adb835d5b7154f96675: Neither blocks nor log segments found. Creating new log.
I20260927 03:31:54.046996 18617 tablet_service.cc:1467] Processing CreateTablet for tablet cbc0471dc63b4a619c0b8d5542ba4e3d (DEFAULT_TABLE table=test-table [id=fe07448b611c402995fa57bd426e30b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:31:54.047343 18617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbc0471dc63b4a619c0b8d5542ba4e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:31:54.047724 18968 log.cc:824] T cbc0471dc63b4a619c0b8d5542ba4e3d P 332386fac6d54adb835d5b7154f96675: Log is configured to *not* fsync() on all Append() calls
I20260927 03:31:54.047902 18969 tablet_bootstrap.cc:492] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f: Bootstrap starting.
I20260927 03:31:54.048527 18968 tablet_bootstrap.cc:492] T cbc0471dc63b4a619c0b8d5542ba4e3d P 332386fac6d54adb835d5b7154f96675: No bootstrap required, opened a new log
I20260927 03:31:54.048574 18969 tablet_bootstrap.cc:654] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f: Neither blocks nor log segments found. Creating new log.
I20260927 03:31:54.048607 18968 ts_tablet_manager.cc:1397] T cbc0471dc63b4a619c0b8d5542ba4e3d P 332386fac6d54adb835d5b7154f96675: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:31:54.049124 18969 log.cc:824] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f: Log is configured to *not* fsync() on all Append() calls
I20260927 03:31:54.050376 18972 tablet_bootstrap.cc:492] T cbc0471dc63b4a619c0b8d5542ba4e3d P eab5841c2f0b4704b85df87735743a94: Bootstrap starting.
I20260927 03:31:54.050426 18968 raft_consensus.cc:348] T cbc0471dc63b4a619c0b8d5542ba4e3d P 332386fac6d54adb835d5b7154f96675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332386fac6d54adb835d5b7154f96675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35755 } } peers { permanent_uuid: "6f640043b4ca4e4986886b6ac0cfd63f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33969 } } peers { permanent_uuid: "eab5841c2f0b4704b85df87735743a94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33319 } }
I20260927 03:31:54.050599 18968 raft_consensus.cc:374] T cbc0471dc63b4a619c0b8d5542ba4e3d P 332386fac6d54adb835d5b7154f96675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:31:54.050644 18968 raft_consensus.cc:729] T cbc0471dc63b4a619c0b8d5542ba4e3d P 332386fac6d54adb835d5b7154f96675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 332386fac6d54adb835d5b7154f96675, State: Initialized, Role: FOLLOWER
I20260927 03:31:54.050787 18968 consensus_queue.cc:260] T cbc0471dc63b4a619c0b8d5542ba4e3d P 332386fac6d54adb835d5b7154f96675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332386fac6d54adb835d5b7154f96675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35755 } } peers { permanent_uuid: "6f640043b4ca4e4986886b6ac0cfd63f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33969 } } peers { permanent_uuid: "eab5841c2f0b4704b85df87735743a94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33319 } }
I20260927 03:31:54.051014 18972 tablet_bootstrap.cc:654] T cbc0471dc63b4a619c0b8d5542ba4e3d P eab5841c2f0b4704b85df87735743a94: Neither blocks nor log segments found. Creating new log.
I20260927 03:31:54.051059 18968 ts_tablet_manager.cc:1428] T cbc0471dc63b4a619c0b8d5542ba4e3d P 332386fac6d54adb835d5b7154f96675: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 03:31:54.051088 18951 heartbeater.cc:503] Master 127.0.98.62:43279 was elected leader, sending a full tablet report...
I20260927 03:31:54.051699 18969 tablet_bootstrap.cc:492] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f: No bootstrap required, opened a new log
I20260927 03:31:54.051728 18972 log.cc:824] T cbc0471dc63b4a619c0b8d5542ba4e3d P eab5841c2f0b4704b85df87735743a94: Log is configured to *not* fsync() on all Append() calls
I20260927 03:31:54.051791 18969 ts_tablet_manager.cc:1397] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 03:31:54.052556 18972 tablet_bootstrap.cc:492] T cbc0471dc63b4a619c0b8d5542ba4e3d P eab5841c2f0b4704b85df87735743a94: No bootstrap required, opened a new log
I20260927 03:31:54.052627 18972 ts_tablet_manager.cc:1397] T cbc0471dc63b4a619c0b8d5542ba4e3d P eab5841c2f0b4704b85df87735743a94: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:31:54.053814 18969 raft_consensus.cc:348] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332386fac6d54adb835d5b7154f96675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35755 } } peers { permanent_uuid: "6f640043b4ca4e4986886b6ac0cfd63f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33969 } } peers { permanent_uuid: "eab5841c2f0b4704b85df87735743a94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33319 } }
I20260927 03:31:54.054008 18969 raft_consensus.cc:374] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:31:54.054045 18969 raft_consensus.cc:729] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f640043b4ca4e4986886b6ac0cfd63f, State: Initialized, Role: FOLLOWER
I20260927 03:31:54.054167 18969 consensus_queue.cc:260] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332386fac6d54adb835d5b7154f96675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35755 } } peers { permanent_uuid: "6f640043b4ca4e4986886b6ac0cfd63f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33969 } } peers { permanent_uuid: "eab5841c2f0b4704b85df87735743a94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33319 } }
I20260927 03:31:54.054440 18969 ts_tablet_manager.cc:1428] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 03:31:54.054461 18817 heartbeater.cc:503] Master 127.0.98.62:43279 was elected leader, sending a full tablet report...
I20260927 03:31:54.054900 18972 raft_consensus.cc:348] T cbc0471dc63b4a619c0b8d5542ba4e3d P eab5841c2f0b4704b85df87735743a94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332386fac6d54adb835d5b7154f96675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35755 } } peers { permanent_uuid: "6f640043b4ca4e4986886b6ac0cfd63f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33969 } } peers { permanent_uuid: "eab5841c2f0b4704b85df87735743a94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33319 } }
I20260927 03:31:54.055070 18972 raft_consensus.cc:374] T cbc0471dc63b4a619c0b8d5542ba4e3d P eab5841c2f0b4704b85df87735743a94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:31:54.055114 18972 raft_consensus.cc:729] T cbc0471dc63b4a619c0b8d5542ba4e3d P eab5841c2f0b4704b85df87735743a94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eab5841c2f0b4704b85df87735743a94, State: Initialized, Role: FOLLOWER
I20260927 03:31:54.055215 18972 consensus_queue.cc:260] T cbc0471dc63b4a619c0b8d5542ba4e3d P eab5841c2f0b4704b85df87735743a94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332386fac6d54adb835d5b7154f96675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35755 } } peers { permanent_uuid: "6f640043b4ca4e4986886b6ac0cfd63f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33969 } } peers { permanent_uuid: "eab5841c2f0b4704b85df87735743a94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33319 } }
I20260927 03:31:54.055528 18683 heartbeater.cc:503] Master 127.0.98.62:43279 was elected leader, sending a full tablet report...
I20260927 03:31:54.055546 18972 ts_tablet_manager.cc:1428] T cbc0471dc63b4a619c0b8d5542ba4e3d P eab5841c2f0b4704b85df87735743a94: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260927 03:31:54.090154 18818 tablet.cc:2369] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:31:54.153887 18975 raft_consensus.cc:482] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:31:54.153909 18976 raft_consensus.cc:482] T cbc0471dc63b4a619c0b8d5542ba4e3d P eab5841c2f0b4704b85df87735743a94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:31:54.154064 18976 raft_consensus.cc:504] T cbc0471dc63b4a619c0b8d5542ba4e3d P eab5841c2f0b4704b85df87735743a94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332386fac6d54adb835d5b7154f96675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35755 } } peers { permanent_uuid: "6f640043b4ca4e4986886b6ac0cfd63f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33969 } } peers { permanent_uuid: "eab5841c2f0b4704b85df87735743a94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33319 } }
I20260927 03:31:54.154065 18975 raft_consensus.cc:504] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332386fac6d54adb835d5b7154f96675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35755 } } peers { permanent_uuid: "6f640043b4ca4e4986886b6ac0cfd63f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33969 } } peers { permanent_uuid: "eab5841c2f0b4704b85df87735743a94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33319 } }
I20260927 03:31:54.154441 18976 leader_election.cc:288] T cbc0471dc63b4a619c0b8d5542ba4e3d P eab5841c2f0b4704b85df87735743a94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 332386fac6d54adb835d5b7154f96675 (127.0.98.3:35755), 6f640043b4ca4e4986886b6ac0cfd63f (127.0.98.2:33969)
I20260927 03:31:54.154759 18975 leader_election.cc:288] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 332386fac6d54adb835d5b7154f96675 (127.0.98.3:35755), eab5841c2f0b4704b85df87735743a94 (127.0.98.1:33319)
I20260927 03:31:54.158490 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.155057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44861 (local address 127.0.98.2:33969)
0927 03:31:54.155094 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:54.155095 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:54.155197 (+ 102us) server_negotiation.cc:407] Connection header received
0927 03:31:54.155286 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:54.155287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:54.155296 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:54.155324 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:54.155533 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.156227 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.157628 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.158145 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.158192 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:54.158203 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:54.158399 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 03:31:54.158426 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:31:54.158502 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.154675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33319 (local address 127.0.98.2:35515)
0927 03:31:54.154904 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:54.154917 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:54.154972 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:54.155219 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:54.155220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:54.155222 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:54.155297 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:54.155298 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.156266 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.156267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:54.157521 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:54.157524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.158246 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.158247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:54.158312 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:54.158315 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:31:54.158365 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 03:31:54.158413 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":49}
I20260927 03:31:54.158654 18965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.155035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35515 (local address 127.0.98.1:33319)
0927 03:31:54.155073 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:54.155075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:54.155083 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:31:54.155107 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:54.155108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:54.155118 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:54.155173 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:54.155370 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.156163 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.157619 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.158152 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.158208 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:54.158217 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:54.158526 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 03:31:54.158566 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:31:54.158665 18964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.154933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38593 (local address 127.0.98.3:35755)
0927 03:31:54.154994 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:54.154997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:54.155006 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:31:54.155088 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:54.155089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:54.155100 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:54.155130 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:54.155374 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.156154 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.157618 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.158264 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.158332 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:54.158343 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:54.158557 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 03:31:54.158595 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":37}
I20260927 03:31:54.158766 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.155033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39603 (local address 127.0.98.3:35755)
0927 03:31:54.155165 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:54.155166 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:54.155172 (+ 6us) server_negotiation.cc:407] Connection header received
0927 03:31:54.155190 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:54.155190 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:54.155199 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:54.155452 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:54.155479 (+ 27us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.156192 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.157615 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.158311 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.158640 (+ 329us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:54.158649 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:54.158665 (+ 16us) server_negotiation.cc:299] Negotiation successful
0927 03:31:54.158698 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
I20260927 03:31:54.158834 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.154736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33969 (local address 127.0.98.1:44861)
0927 03:31:54.155146 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:54.155157 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:54.155218 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:54.155379 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:54.155380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:54.155382 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:54.155457 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:54.155459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.156314 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.156315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:54.157521 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:54.157524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.158258 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.158259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:54.158320 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:54.158324 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:31:54.158718 (+ 394us) client_negotiation.cc:240] Negotiation successful
0927 03:31:54.158763 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":99,"threads_started":1}
I20260927 03:31:54.158910 18977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.154689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35755 (local address 127.0.98.2:38593)
0927 03:31:54.154948 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:54.154957 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:54.155003 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:54.155209 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:54.155209 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:54.155212 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:54.155296 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:54.155298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.156256 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.156256 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:54.157512 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:54.157514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.158378 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.158379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:54.158458 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:54.158475 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 03:31:54.158775 (+ 300us) client_negotiation.cc:240] Negotiation successful
0927 03:31:54.158825 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":3,"thread_start_us":124,"threads_started":1}
I20260927 03:31:54.158897 18637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc0471dc63b4a619c0b8d5542ba4e3d" candidate_uuid: "6f640043b4ca4e4986886b6ac0cfd63f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eab5841c2f0b4704b85df87735743a94" is_pre_election: true
I20260927 03:31:54.159015 18637 raft_consensus.cc:2454] T cbc0471dc63b4a619c0b8d5542ba4e3d P eab5841c2f0b4704b85df87735743a94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f640043b4ca4e4986886b6ac0cfd63f in term 0.
I20260927 03:31:54.159132 18771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc0471dc63b4a619c0b8d5542ba4e3d" candidate_uuid: "eab5841c2f0b4704b85df87735743a94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f640043b4ca4e4986886b6ac0cfd63f" is_pre_election: true
I20260927 03:31:54.159197 18706 leader_election.cc:302] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f640043b4ca4e4986886b6ac0cfd63f, eab5841c2f0b4704b85df87735743a94; no voters:
I20260927 03:31:54.159225 18771 raft_consensus.cc:2454] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eab5841c2f0b4704b85df87735743a94 in term 0.
I20260927 03:31:54.159219 18905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc0471dc63b4a619c0b8d5542ba4e3d" candidate_uuid: "6f640043b4ca4e4986886b6ac0cfd63f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "332386fac6d54adb835d5b7154f96675" is_pre_election: true
I20260927 03:31:54.159313 18975 raft_consensus.cc:2781] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:31:54.159358 18975 raft_consensus.cc:482] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:31:54.159380 18975 raft_consensus.cc:3037] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:31:54.159382 18905 raft_consensus.cc:2454] T cbc0471dc63b4a619c0b8d5542ba4e3d P 332386fac6d54adb835d5b7154f96675 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f640043b4ca4e4986886b6ac0cfd63f in term 0.
I20260927 03:31:54.159389 18573 leader_election.cc:302] T cbc0471dc63b4a619c0b8d5542ba4e3d P eab5841c2f0b4704b85df87735743a94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f640043b4ca4e4986886b6ac0cfd63f, eab5841c2f0b4704b85df87735743a94; no voters:
I20260927 03:31:54.158902 18685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.154643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35755 (local address 127.0.98.1:39603)
0927 03:31:54.154725 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:54.154739 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:54.154821 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:54.155292 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:54.155293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:54.155294 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:54.155361 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:54.155363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.156281 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.156283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:54.157505 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:54.157508 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.158396 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.158397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:54.158478 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:54.158481 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:31:54.158510 (+ 29us) client_negotiation.cc:240] Negotiation successful
0927 03:31:54.158557 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":6}
I20260927 03:31:54.159729 18976 raft_consensus.cc:2781] T cbc0471dc63b4a619c0b8d5542ba4e3d P eab5841c2f0b4704b85df87735743a94 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:31:54.159785 18976 raft_consensus.cc:482] T cbc0471dc63b4a619c0b8d5542ba4e3d P eab5841c2f0b4704b85df87735743a94 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:31:54.159749 18905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc0471dc63b4a619c0b8d5542ba4e3d" candidate_uuid: "eab5841c2f0b4704b85df87735743a94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "332386fac6d54adb835d5b7154f96675" is_pre_election: true
I20260927 03:31:54.159824 18976 raft_consensus.cc:3037] T cbc0471dc63b4a619c0b8d5542ba4e3d P eab5841c2f0b4704b85df87735743a94 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:31:54.159834 18905 raft_consensus.cc:2454] T cbc0471dc63b4a619c0b8d5542ba4e3d P 332386fac6d54adb835d5b7154f96675 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eab5841c2f0b4704b85df87735743a94 in term 0.
I20260927 03:31:54.160362 18975 raft_consensus.cc:504] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332386fac6d54adb835d5b7154f96675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35755 } } peers { permanent_uuid: "6f640043b4ca4e4986886b6ac0cfd63f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33969 } } peers { permanent_uuid: "eab5841c2f0b4704b85df87735743a94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33319 } }
I20260927 03:31:54.160584 18975 leader_election.cc:288] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f [CANDIDATE]: Term 1 election: Requested vote from peers 332386fac6d54adb835d5b7154f96675 (127.0.98.3:35755), eab5841c2f0b4704b85df87735743a94 (127.0.98.1:33319)
I20260927 03:31:54.160634 18905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc0471dc63b4a619c0b8d5542ba4e3d" candidate_uuid: "6f640043b4ca4e4986886b6ac0cfd63f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "332386fac6d54adb835d5b7154f96675"
I20260927 03:31:54.160722 18905 raft_consensus.cc:3037] T cbc0471dc63b4a619c0b8d5542ba4e3d P 332386fac6d54adb835d5b7154f96675 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:31:54.160730 18976 raft_consensus.cc:504] T cbc0471dc63b4a619c0b8d5542ba4e3d P eab5841c2f0b4704b85df87735743a94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332386fac6d54adb835d5b7154f96675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35755 } } peers { permanent_uuid: "6f640043b4ca4e4986886b6ac0cfd63f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33969 } } peers { permanent_uuid: "eab5841c2f0b4704b85df87735743a94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33319 } }
I20260927 03:31:54.160756 18637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc0471dc63b4a619c0b8d5542ba4e3d" candidate_uuid: "6f640043b4ca4e4986886b6ac0cfd63f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eab5841c2f0b4704b85df87735743a94"
I20260927 03:31:54.160841 18637 raft_consensus.cc:2379] T cbc0471dc63b4a619c0b8d5542ba4e3d P eab5841c2f0b4704b85df87735743a94 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6f640043b4ca4e4986886b6ac0cfd63f in current term 1: Already voted for candidate eab5841c2f0b4704b85df87735743a94 in this term.
I20260927 03:31:54.160866 18976 leader_election.cc:288] T cbc0471dc63b4a619c0b8d5542ba4e3d P eab5841c2f0b4704b85df87735743a94 [CANDIDATE]: Term 1 election: Requested vote from peers 332386fac6d54adb835d5b7154f96675 (127.0.98.3:35755), 6f640043b4ca4e4986886b6ac0cfd63f (127.0.98.2:33969)
I20260927 03:31:54.161020 18771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc0471dc63b4a619c0b8d5542ba4e3d" candidate_uuid: "eab5841c2f0b4704b85df87735743a94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f640043b4ca4e4986886b6ac0cfd63f"
I20260927 03:31:54.161020 18904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc0471dc63b4a619c0b8d5542ba4e3d" candidate_uuid: "eab5841c2f0b4704b85df87735743a94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "332386fac6d54adb835d5b7154f96675"
I20260927 03:31:54.161099 18771 raft_consensus.cc:2379] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate eab5841c2f0b4704b85df87735743a94 in current term 1: Already voted for candidate 6f640043b4ca4e4986886b6ac0cfd63f in this term.
I20260927 03:31:54.161309 18573 leader_election.cc:302] T cbc0471dc63b4a619c0b8d5542ba4e3d P eab5841c2f0b4704b85df87735743a94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eab5841c2f0b4704b85df87735743a94; no voters: 332386fac6d54adb835d5b7154f96675, 6f640043b4ca4e4986886b6ac0cfd63f
I20260927 03:31:54.161423 18976 raft_consensus.cc:2726] T cbc0471dc63b4a619c0b8d5542ba4e3d P eab5841c2f0b4704b85df87735743a94 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 03:31:54.161638 18905 raft_consensus.cc:2454] T cbc0471dc63b4a619c0b8d5542ba4e3d P 332386fac6d54adb835d5b7154f96675 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f640043b4ca4e4986886b6ac0cfd63f in term 1.
I20260927 03:31:54.161790 18707 leader_election.cc:302] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 332386fac6d54adb835d5b7154f96675, 6f640043b4ca4e4986886b6ac0cfd63f; no voters: eab5841c2f0b4704b85df87735743a94
I20260927 03:31:54.161895 18975 raft_consensus.cc:2781] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:31:54.162045 18975 raft_consensus.cc:686] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f [term 1 LEADER]: Becoming Leader. State: Replica: 6f640043b4ca4e4986886b6ac0cfd63f, State: Running, Role: LEADER
I20260927 03:31:54.162140 18975 consensus_queue.cc:237] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332386fac6d54adb835d5b7154f96675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35755 } } peers { permanent_uuid: "6f640043b4ca4e4986886b6ac0cfd63f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33969 } } peers { permanent_uuid: "eab5841c2f0b4704b85df87735743a94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33319 } }
I20260927 03:31:54.162951 18492 catalog_manager.cc:5495] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f640043b4ca4e4986886b6ac0cfd63f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6f640043b4ca4e4986886b6ac0cfd63f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "332386fac6d54adb835d5b7154f96675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f640043b4ca4e4986886b6ac0cfd63f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eab5841c2f0b4704b85df87735743a94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33319 } health_report { overall_health: UNKNOWN } } }
W20260927 03:31:54.184712 18684 tablet.cc:2369] T cbc0471dc63b4a619c0b8d5542ba4e3d P eab5841c2f0b4704b85df87735743a94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:31:54.188120 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.185592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60462 (local address 127.0.98.2:33969)
0927 03:31:54.185647 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:54.185649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:54.185658 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:31:54.185705 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:54.185710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:54.185734 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:54.185813 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:31:54.185972 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.186445 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.187237 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.187490 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.187582 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:54.187731 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:31:54.187838 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:31:54.187952 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:54.188001 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 03:31:54.188012 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 03:31:54.189944 18637 raft_consensus.cc:1261] T cbc0471dc63b4a619c0b8d5542ba4e3d P eab5841c2f0b4704b85df87735743a94 [term 1 FOLLOWER]: Refusing update from remote peer 6f640043b4ca4e4986886b6ac0cfd63f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:31:54.190017 18905 raft_consensus.cc:1261] T cbc0471dc63b4a619c0b8d5542ba4e3d P 332386fac6d54adb835d5b7154f96675 [term 1 FOLLOWER]: Refusing update from remote peer 6f640043b4ca4e4986886b6ac0cfd63f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:31:54.190270 18980 consensus_queue.cc:1035] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eab5841c2f0b4704b85df87735743a94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:31:54.190440 18975 consensus_queue.cc:1035] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f [LEADER]: Connected to new peer: Peer: permanent_uuid: "332386fac6d54adb835d5b7154f96675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:31:54.193440 18985 mvcc.cc:205] Tried to move back new op lower bound from 7333805728519163904 to 7333805728408633344. Current Snapshot: MvccSnapshot[applied={T|T < 7333805728519163904}]
I20260927 03:31:54.193960 18984 mvcc.cc:205] Tried to move back new op lower bound from 7333805728519163904 to 7333805728408633344. Current Snapshot: MvccSnapshot[applied={T|T < 7333805728519163904}]
I20260927 03:31:54.195497 18492 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:31:54.195590 18492 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:31:54.196355 18492 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3b8fac51c4d4dfca5c2e755a47d75db: Sending DeleteTablet for 3 replicas of tablet cbc0471dc63b4a619c0b8d5542ba4e3d
I20260927 03:31:54.196738 18751 tablet_service.cc:1514] Processing DeleteTablet for tablet cbc0471dc63b4a619c0b8d5542ba4e3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:31:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60460
I20260927 03:31:54.196843 18885 tablet_service.cc:1514] Processing DeleteTablet for tablet cbc0471dc63b4a619c0b8d5542ba4e3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:31:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37896
I20260927 03:31:54.197000 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 18554
I20260927 03:31:54.197052 18617 tablet_service.cc:1514] Processing DeleteTablet for tablet cbc0471dc63b4a619c0b8d5542ba4e3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:31:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38410
I20260927 03:31:54.197880 18994 tablet_replica.cc:330] T cbc0471dc63b4a619c0b8d5542ba4e3d P 332386fac6d54adb835d5b7154f96675: stopping tablet replica
I20260927 03:31:54.197976 18994 raft_consensus.cc:2229] T cbc0471dc63b4a619c0b8d5542ba4e3d P 332386fac6d54adb835d5b7154f96675 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:31:54.198073 18994 raft_consensus.cc:2258] T cbc0471dc63b4a619c0b8d5542ba4e3d P 332386fac6d54adb835d5b7154f96675 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:31:54.198359 18993 tablet_replica.cc:330] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f: stopping tablet replica
I20260927 03:31:54.198505 18993 raft_consensus.cc:2229] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:31:54.198545 18994 ts_tablet_manager.cc:1905] T cbc0471dc63b4a619c0b8d5542ba4e3d P 332386fac6d54adb835d5b7154f96675: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:31:54.198678 18993 raft_consensus.cc:2258] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:31:54.199151 18993 ts_tablet_manager.cc:1905] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:31:54.200706 18994 ts_tablet_manager.cc:1918] T cbc0471dc63b4a619c0b8d5542ba4e3d P 332386fac6d54adb835d5b7154f96675: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:31:54.200798 18994 log.cc:1196] T cbc0471dc63b4a619c0b8d5542ba4e3d P 332386fac6d54adb835d5b7154f96675: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-2/wal/wals/cbc0471dc63b4a619c0b8d5542ba4e3d
I20260927 03:31:54.201126 18994 ts_tablet_manager.cc:1939] T cbc0471dc63b4a619c0b8d5542ba4e3d P 332386fac6d54adb835d5b7154f96675: Deleting consensus metadata
I20260927 03:31:54.201294 18993 ts_tablet_manager.cc:1918] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:31:54.201417 18993 log.cc:1196] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790479675865103-392-0/minicluster-data/ts-1/wal/wals/cbc0471dc63b4a619c0b8d5542ba4e3d
I20260927 03:31:54.201609 18479 catalog_manager.cc:4841] TS 332386fac6d54adb835d5b7154f96675 (127.0.98.3:35755): tablet cbc0471dc63b4a619c0b8d5542ba4e3d (table test-table [id=fe07448b611c402995fa57bd426e30b3]) successfully deleted
I20260927 03:31:54.201743 18993 ts_tablet_manager.cc:1939] T cbc0471dc63b4a619c0b8d5542ba4e3d P 6f640043b4ca4e4986886b6ac0cfd63f: Deleting consensus metadata
I20260927 03:31:54.202135 18479 catalog_manager.cc:4841] TS 6f640043b4ca4e4986886b6ac0cfd63f (127.0.98.2:33969): tablet cbc0471dc63b4a619c0b8d5542ba4e3d (table test-table [id=fe07448b611c402995fa57bd426e30b3]) successfully deleted
W20260927 03:31:54.204418 18478 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33319 (error 108)
I20260927 03:31:54.204692 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 18688
I20260927 03:31:54.205363 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.204800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33319 (local address 127.0.0.1:38412)
0927 03:31:54.205293 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:54.205334 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475}
W20260927 03:31:54.205504 18478 catalog_manager.cc:4584] TS eab5841c2f0b4704b85df87735743a94 (127.0.98.1:33319): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cbc0471dc63b4a619c0b8d5542ba4e3d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33319: connect: Connection refused (error 111)
I20260927 03:31:54.210776 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 18822
I20260927 03:31:54.215215 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 18460
2026-09-27T03:31:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2055 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T03:31:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:31:54Z Disabled control of system clock
I20260927 03:31:54.242484 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33871
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 03:31:54.330703 18999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:54.330955 18999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:54.330977 18999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:54.332173 18999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:31:54.332219 18999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:54.332235 18999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:54.332249 18999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:31:54.332263 18999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:31:54.333564 18999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33871
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:54.333781 18999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:54.334039 18999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:31:54.339061 19007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:54.339093 19005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:54.339299 18999 server_base.cc:1034] running on GCE node
W20260927 03:31:54.339061 19004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:54.339673 18999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:54.340021 18999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:54.341203 18999 hybrid_clock.cc:648] HybridClock initialized: now 1790479914341181 us; error 37 us; skew 500 ppm
I20260927 03:31:54.342478 18999 webserver.cc:460] Webserver started at http://127.0.98.62:33207/ using document root <none> and password file <none>
I20260927 03:31:54.342797 18999 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:54.342875 18999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:54.343045 18999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:31:54.344197 18999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "2d08b17bab784b7680c142b24a4c133d"
format_stamp: "Formatted at 2026-09-27 03:31:54 on dist-test-slave-j0hf"
server_key: "c001f00998e056d15b3254fa6083607c"
server_key_iv: "da65b2efceef2e21cd763e0db0b0123a"
server_key_version: "encryptionkey@0"
I20260927 03:31:54.344569 18999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "2d08b17bab784b7680c142b24a4c133d"
format_stamp: "Formatted at 2026-09-27 03:31:54 on dist-test-slave-j0hf"
server_key: "c001f00998e056d15b3254fa6083607c"
server_key_iv: "da65b2efceef2e21cd763e0db0b0123a"
server_key_version: "encryptionkey@0"
I20260927 03:31:54.345867 18999 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:31:54.346649 19013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:54.346854 18999 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:31:54.346976 18999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "2d08b17bab784b7680c142b24a4c133d"
format_stamp: "Formatted at 2026-09-27 03:31:54 on dist-test-slave-j0hf"
server_key: "c001f00998e056d15b3254fa6083607c"
server_key_iv: "da65b2efceef2e21cd763e0db0b0123a"
server_key_version: "encryptionkey@0"
I20260927 03:31:54.347057 18999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:54.362565 18999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:54.362929 18999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:54.363098 18999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:54.367347 18999 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33871
I20260927 03:31:54.367381 19065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33871 every 8 connection(s)
I20260927 03:31:54.367822 18999 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:31:54.368825 19066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:31:54.371161 19066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d08b17bab784b7680c142b24a4c133d: Bootstrap starting.
I20260927 03:31:54.371845 19066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d08b17bab784b7680c142b24a4c133d: Neither blocks nor log segments found. Creating new log.
I20260927 03:31:54.372159 19066 log.cc:824] T 00000000000000000000000000000000 P 2d08b17bab784b7680c142b24a4c133d: Log is configured to *not* fsync() on all Append() calls
I20260927 03:31:54.372845 19066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d08b17bab784b7680c142b24a4c133d: No bootstrap required, opened a new log
I20260927 03:31:54.374444 19066 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d08b17bab784b7680c142b24a4c133d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d08b17bab784b7680c142b24a4c133d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33871 } }
I20260927 03:31:54.374622 19066 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d08b17bab784b7680c142b24a4c133d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:31:54.374670 19066 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d08b17bab784b7680c142b24a4c133d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d08b17bab784b7680c142b24a4c133d, State: Initialized, Role: FOLLOWER
I20260927 03:31:54.374832 19066 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d08b17bab784b7680c142b24a4c133d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d08b17bab784b7680c142b24a4c133d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33871 } }
I20260927 03:31:54.374919 19066 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d08b17bab784b7680c142b24a4c133d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:31:54.374974 19066 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d08b17bab784b7680c142b24a4c133d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:31:54.375025 19066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d08b17bab784b7680c142b24a4c133d [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:31:54.375586 19066 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d08b17bab784b7680c142b24a4c133d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d08b17bab784b7680c142b24a4c133d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33871 } }
I20260927 03:31:54.375695 19066 leader_election.cc:302] T 00000000000000000000000000000000 P 2d08b17bab784b7680c142b24a4c133d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d08b17bab784b7680c142b24a4c133d; no voters:
I20260927 03:31:54.375898 19066 leader_election.cc:288] T 00000000000000000000000000000000 P 2d08b17bab784b7680c142b24a4c133d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:31:54.375975 19069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d08b17bab784b7680c142b24a4c133d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:31:54.376163 19066 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d08b17bab784b7680c142b24a4c133d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:31:54.376195 19069 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d08b17bab784b7680c142b24a4c133d [term 1 LEADER]: Becoming Leader. State: Replica: 2d08b17bab784b7680c142b24a4c133d, State: Running, Role: LEADER
I20260927 03:31:54.376426 19069 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d08b17bab784b7680c142b24a4c133d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d08b17bab784b7680c142b24a4c133d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33871 } }
I20260927 03:31:54.376781 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 18999
I20260927 03:31:54.376909 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:31:54.376955 19070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d08b17bab784b7680c142b24a4c133d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d08b17bab784b7680c142b24a4c133d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d08b17bab784b7680c142b24a4c133d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33871 } } }
I20260927 03:31:54.377025 19070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d08b17bab784b7680c142b24a4c133d [sys.catalog]: This master's current role is: LEADER
I20260927 03:31:54.377168 392 external_mini_cluster.cc:1428] Setting key ea2bda23b2ca7cfb71187ed04aa94a56
I20260927 03:31:54.377192 19071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d08b17bab784b7680c142b24a4c133d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d08b17bab784b7680c142b24a4c133d. Latest consensus state: current_term: 1 leader_uuid: "2d08b17bab784b7680c142b24a4c133d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d08b17bab784b7680c142b24a4c133d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33871 } } }
I20260927 03:31:54.377241 19071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d08b17bab784b7680c142b24a4c133d [sys.catalog]: This master's current role is: LEADER
I20260927 03:31:54.377267 19078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:31:54.377768 19078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:31:54.379513 19078 catalog_manager.cc:1347] Generated new cluster ID: 7e335f2124ce48e38d72765c504c5707
I20260927 03:31:54.379566 19078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:31:54.383083 19088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.377934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41952 (local address 127.0.98.62:33871)
0927 03:31:54.378377 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:54.378384 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:54.378407 (+ 23us) server_negotiation.cc:407] Connection header received
0927 03:31:54.378501 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:54.378507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:54.378603 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:54.378736 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:54.379164 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.380223 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.381035 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.381516 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.381648 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:54.382377 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:54.382381 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:54.382381 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:31:54.382563 (+ 182us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:54.382594 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:31:54.382605 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:54.382681 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:54.382756 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 03:31:54.382922 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":94,"threads_started":1}
I20260927 03:31:54.387796 19078 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:31:54.388438 19078 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:31:54.393005 19078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d08b17bab784b7680c142b24a4c133d: Generated new TSK 0
I20260927 03:31:54.393190 19078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:31:54.398121 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33871
--builtin_ntp_servers=127.0.98.20:37877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 03:31:54.480006 19090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:54.480227 19090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:54.480262 19090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:54.481391 19090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:54.481452 19090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:54.481490 19090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:31:54.482900 19090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:54.483162 19090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:54.483515 19090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:31:54.489079 19095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:54.489176 19096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:54.489300 19090 server_base.cc:1034] running on GCE node
W20260927 03:31:54.489136 19098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:54.489511 19090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:54.489764 19090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:54.490923 19090 hybrid_clock.cc:648] HybridClock initialized: now 1790479914490911 us; error 29 us; skew 500 ppm
I20260927 03:31:54.492249 19090 webserver.cc:460] Webserver started at http://127.0.98.1:33719/ using document root <none> and password file <none>
I20260927 03:31:54.492509 19090 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:54.492600 19090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:54.492748 19090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:31:54.493783 19090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-0/data/instance:
uuid: "f144a169f5494497ba00a3a134fc6ffb"
format_stamp: "Formatted at 2026-09-27 03:31:54 on dist-test-slave-j0hf"
server_key: "983514d24156cbe905c348a0d8bf99be"
server_key_iv: "12f2ad5a1efd6347ba02e6cf838a0b82"
server_key_version: "encryptionkey@0"
I20260927 03:31:54.494146 19090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f144a169f5494497ba00a3a134fc6ffb"
format_stamp: "Formatted at 2026-09-27 03:31:54 on dist-test-slave-j0hf"
server_key: "983514d24156cbe905c348a0d8bf99be"
server_key_iv: "12f2ad5a1efd6347ba02e6cf838a0b82"
server_key_version: "encryptionkey@0"
I20260927 03:31:54.495554 19090 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:31:54.496325 19104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:54.496508 19090 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:31:54.496613 19090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "f144a169f5494497ba00a3a134fc6ffb"
format_stamp: "Formatted at 2026-09-27 03:31:54 on dist-test-slave-j0hf"
server_key: "983514d24156cbe905c348a0d8bf99be"
server_key_iv: "12f2ad5a1efd6347ba02e6cf838a0b82"
server_key_version: "encryptionkey@0"
I20260927 03:31:54.496701 19090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:54.506199 19090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:54.506479 19090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:54.506623 19090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:54.506829 19090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:31:54.507128 19090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:31:54.507181 19090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:54.507220 19090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:31:54.507247 19090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:54.512820 19090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44823
I20260927 03:31:54.512873 19217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44823 every 8 connection(s)
I20260927 03:31:54.513249 19090 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:31:54.518131 19088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.513981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32825 (local address 127.0.98.62:33871)
0927 03:31:54.514048 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:54.514050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:54.514256 (+ 206us) server_negotiation.cc:407] Connection header received
0927 03:31:54.514730 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:54.514731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:54.514741 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:54.514795 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:54.515466 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.515958 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.516887 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.517072 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.517121 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:54.517932 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:54.517933 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:54.517933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:31:54.517938 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:54.517948 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:31:54.517951 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:54.517989 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:54.518053 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 03:31:54.518075 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 03:31:54.518360 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.513938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33871 (local address 127.0.98.1:32825)
0927 03:31:54.514127 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:54.514168 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:54.514648 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:54.514854 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:54.514857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:54.514865 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:31:54.515351 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:54.515356 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.516039 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.516039 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:54.516811 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:54.516813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.517141 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.517141 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:54.517244 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:54.517801 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:31:54.517802 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:31:54.517834 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:31:54.517857 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:31:54.517990 (+ 133us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:31:54.517992 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:31:54.517994 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:31:54.518041 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 03:31:54.518116 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
I20260927 03:31:54.518945 19218 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33871
I20260927 03:31:54.519035 19218 heartbeater.cc:464] Registering TS with master...
I20260927 03:31:54.519172 19218 heartbeater.cc:511] Master 127.0.98.62:33871 requested a full tablet report, sending...
I20260927 03:31:54.519670 19030 ts_manager.cc:194] Registered new tserver with Master: f144a169f5494497ba00a3a134fc6ffb (127.0.98.1:44823)
I20260927 03:31:54.520227 19030 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:32825
I20260927 03:31:54.523159 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 19090
I20260927 03:31:54.523264 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:31:54.523419 392 external_mini_cluster.cc:1428] Setting key b21f3ef86b7ce1c32fe9628af295b394
I20260927 03:31:54.524334 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33871
--builtin_ntp_servers=127.0.98.20:37877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 03:31:54.604487 19221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:54.604785 19221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:54.604825 19221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:54.605975 19221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:54.606041 19221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:54.606086 19221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:31:54.607402 19221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:54.607717 19221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:54.608016 19221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:31:54.612849 19227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:54.612816 19226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:54.612818 19229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:54.613040 19221 server_base.cc:1034] running on GCE node
I20260927 03:31:54.613250 19221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:54.613490 19221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:54.614624 19221 hybrid_clock.cc:648] HybridClock initialized: now 1790479914614603 us; error 31 us; skew 500 ppm
I20260927 03:31:54.621500 19221 webserver.cc:460] Webserver started at http://127.0.98.2:43819/ using document root <none> and password file <none>
I20260927 03:31:54.621766 19221 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:54.621819 19221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:54.622012 19221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:31:54.623143 19221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-1/data/instance:
uuid: "2da4a91396f54b859ed2e693af576067"
format_stamp: "Formatted at 2026-09-27 03:31:54 on dist-test-slave-j0hf"
server_key: "5ccc6f8de92c5385be6758ed9bf12d74"
server_key_iv: "e48aef636a7bbfff07604cc1cc7ad54e"
server_key_version: "encryptionkey@0"
I20260927 03:31:54.623545 19221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2da4a91396f54b859ed2e693af576067"
format_stamp: "Formatted at 2026-09-27 03:31:54 on dist-test-slave-j0hf"
server_key: "5ccc6f8de92c5385be6758ed9bf12d74"
server_key_iv: "e48aef636a7bbfff07604cc1cc7ad54e"
server_key_version: "encryptionkey@0"
I20260927 03:31:54.625281 19221 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 03:31:54.626070 19235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:54.626250 19221 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:31:54.626349 19221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "2da4a91396f54b859ed2e693af576067"
format_stamp: "Formatted at 2026-09-27 03:31:54 on dist-test-slave-j0hf"
server_key: "5ccc6f8de92c5385be6758ed9bf12d74"
server_key_iv: "e48aef636a7bbfff07604cc1cc7ad54e"
server_key_version: "encryptionkey@0"
I20260927 03:31:54.626438 19221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:54.651499 19221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:54.651839 19221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:54.652004 19221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:54.652242 19221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:31:54.652601 19221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:31:54.652657 19221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:54.652699 19221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:31:54.652729 19221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:54.658464 19221 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41153
I20260927 03:31:54.658521 19348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41153 every 8 connection(s)
I20260927 03:31:54.658869 19221 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:31:54.664088 19088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.659714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42181 (local address 127.0.98.62:33871)
0927 03:31:54.659777 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:54.659779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:54.660013 (+ 234us) server_negotiation.cc:407] Connection header received
0927 03:31:54.660499 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:54.660500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:54.660509 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:54.660569 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:54.661234 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.661728 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.662738 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.662918 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.662974 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:54.663841 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:54.663842 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:54.663843 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:31:54.663847 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:54.663861 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:31:54.663865 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:54.663910 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:54.663995 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 03:31:54.664025 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 03:31:54.664323 19351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.659673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33871 (local address 127.0.98.2:42181)
0927 03:31:54.659874 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:54.659916 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:54.660406 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:54.660634 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:54.660638 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:54.660646 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:31:54.661159 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:54.661164 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.661825 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.661826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:54.662641 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:54.662643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.662998 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.662999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:54.663112 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:54.663700 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:31:54.663700 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:31:54.663733 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:31:54.663757 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:31:54.663923 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:31:54.663927 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:31:54.663930 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:31:54.663983 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 03:31:54.664070 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":98,"threads_started":1}
I20260927 03:31:54.664925 19349 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33871
I20260927 03:31:54.665035 19349 heartbeater.cc:464] Registering TS with master...
I20260927 03:31:54.665227 19349 heartbeater.cc:511] Master 127.0.98.62:33871 requested a full tablet report, sending...
I20260927 03:31:54.665661 19030 ts_manager.cc:194] Registered new tserver with Master: 2da4a91396f54b859ed2e693af576067 (127.0.98.2:41153)
I20260927 03:31:54.666009 19030 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:42181
I20260927 03:31:54.669003 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 19221
I20260927 03:31:54.669097 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:31:54.669252 392 external_mini_cluster.cc:1428] Setting key 76e645a7c30679af944d72c7b1db075e
I20260927 03:31:54.670130 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33871
--builtin_ntp_servers=127.0.98.20:37877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 03:31:54.749478 19352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:54.749702 19352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:54.749725 19352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:54.750984 19352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:54.751077 19352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:54.751143 19352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:31:54.752593 19352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:54.752892 19352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:54.753225 19352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:31:54.758239 19360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:54.758308 19358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:54.758495 19352 server_base.cc:1034] running on GCE node
W20260927 03:31:54.758234 19357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:54.758761 19352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:54.758975 19352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:54.760114 19352 hybrid_clock.cc:648] HybridClock initialized: now 1790479914760103 us; error 26 us; skew 500 ppm
I20260927 03:31:54.761423 19352 webserver.cc:460] Webserver started at http://127.0.98.3:37011/ using document root <none> and password file <none>
I20260927 03:31:54.761631 19352 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:54.761674 19352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:54.761763 19352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:31:54.762771 19352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-2/data/instance:
uuid: "3b3fe43b2125438fb10972ff4cb1dc55"
format_stamp: "Formatted at 2026-09-27 03:31:54 on dist-test-slave-j0hf"
server_key: "c820b3bc605464ad8e0c02d8ab088550"
server_key_iv: "25200e79148c00da7caf0b95549d9b10"
server_key_version: "encryptionkey@0"
I20260927 03:31:54.763072 19352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3b3fe43b2125438fb10972ff4cb1dc55"
format_stamp: "Formatted at 2026-09-27 03:31:54 on dist-test-slave-j0hf"
server_key: "c820b3bc605464ad8e0c02d8ab088550"
server_key_iv: "25200e79148c00da7caf0b95549d9b10"
server_key_version: "encryptionkey@0"
I20260927 03:31:54.764468 19352 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:31:54.765127 19366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:54.765300 19352 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:31:54.765403 19352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "3b3fe43b2125438fb10972ff4cb1dc55"
format_stamp: "Formatted at 2026-09-27 03:31:54 on dist-test-slave-j0hf"
server_key: "c820b3bc605464ad8e0c02d8ab088550"
server_key_iv: "25200e79148c00da7caf0b95549d9b10"
server_key_version: "encryptionkey@0"
I20260927 03:31:54.765478 19352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:54.780311 19352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:54.780610 19352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:54.780759 19352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:54.780987 19352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:31:54.781309 19352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:31:54.781363 19352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:54.781397 19352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:31:54.781423 19352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:54.787037 19352 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35037
I20260927 03:31:54.787076 19479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35037 every 8 connection(s)
I20260927 03:31:54.787408 19352 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:31:54.792349 19088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.788184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47281 (local address 127.0.98.62:33871)
0927 03:31:54.788245 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:54.788247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:54.788483 (+ 236us) server_negotiation.cc:407] Connection header received
0927 03:31:54.788938 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:54.788939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:54.788949 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:54.789011 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:54.789652 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.790138 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.791075 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.791253 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.791303 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:54.792119 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:54.792121 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:54.792121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:31:54.792125 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:54.792139 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:31:54.792143 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:54.792187 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:54.792267 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 03:31:54.792291 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:31:54.792567 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.788146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33871 (local address 127.0.98.3:47281)
0927 03:31:54.788345 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:54.788384 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:54.788856 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:54.789065 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:54.789069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:54.789077 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:31:54.789580 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:54.789584 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.790225 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.790225 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:54.790993 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:54.790995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.791329 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.791330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:54.791481 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:54.791989 (+ 508us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:31:54.791990 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:31:54.792017 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:31:54.792040 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:31:54.792198 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:31:54.792202 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:31:54.792204 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:31:54.792255 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 03:31:54.792332 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":103,"threads_started":1}
I20260927 03:31:54.793040 19480 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33871
I20260927 03:31:54.793157 19480 heartbeater.cc:464] Registering TS with master...
I20260927 03:31:54.793327 19480 heartbeater.cc:511] Master 127.0.98.62:33871 requested a full tablet report, sending...
I20260927 03:31:54.793664 19030 ts_manager.cc:194] Registered new tserver with Master: 3b3fe43b2125438fb10972ff4cb1dc55 (127.0.98.3:35037)
I20260927 03:31:54.794023 19030 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:47281
I20260927 03:31:54.794926 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 19352
I20260927 03:31:54.795044 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:31:54.795221 392 external_mini_cluster.cc:1428] Setting key e20a99964a7e4e87a42628f28122af7a
I20260927 03:31:54.796020 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:31:54.799988 19088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.797321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41960 (local address 127.0.98.62:33871)
0927 03:31:54.797372 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:54.797374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:54.797515 (+ 141us) server_negotiation.cc:407] Connection header received
0927 03:31:54.797564 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:54.797565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:54.797574 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:54.797622 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:54.797761 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.798199 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.798887 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.799056 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.799103 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:54.799772 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:54.799773 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:54.799773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:31:54.799777 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:54.799789 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:31:54.799792 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:54.799866 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:54.799924 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 03:31:54.799933 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 03:31:54.801496 19030 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:31:54.803066 19030 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:31:54.809005 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.805187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35037 (local address 127.0.0.1:56478)
0927 03:31:54.805340 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:54.805375 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:54.805439 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:54.805716 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:54.805720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:54.805722 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:54.805786 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:54.805787 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.806796 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.806797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:54.807836 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:54.807839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.808725 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.808725 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:54.808801 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:54.808813 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 03:31:54.808886 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 03:31:54.808934 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
I20260927 03:31:54.809017 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.805359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56478 (local address 127.0.98.3:35037)
0927 03:31:54.805497 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:54.805501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:54.805511 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:31:54.805552 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:54.805554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:54.805570 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:54.805648 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:54.805866 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.806713 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.807930 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.808647 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.808724 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:54.808811 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:54.808898 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 03:31:54.808933 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":67,"threads_started":1}
I20260927 03:31:54.809274 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.805261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43210 (local address 127.0.98.1:44823)
0927 03:31:54.805429 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:54.805434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:54.805587 (+ 153us) server_negotiation.cc:407] Connection header received
0927 03:31:54.805708 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:54.805710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:54.805725 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:54.805775 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:54.805970 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.807083 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.808219 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.808809 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.808875 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:54.808951 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:54.809153 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 03:31:54.809191 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":65,"threads_started":1}
I20260927 03:31:54.809495 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.805191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44823 (local address 127.0.0.1:43210)
0927 03:31:54.805518 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:54.805529 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:54.805616 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:54.805860 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:54.805861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:54.805862 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:54.805892 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:54.805893 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.807180 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.807181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:54.808128 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:54.808131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.808997 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.808998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:54.809061 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:54.809064 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:31:54.809386 (+ 322us) client_negotiation.cc:240] Negotiation successful
0927 03:31:54.809430 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":7,"thread_start_us":95,"threads_started":1}
I20260927 03:31:54.809870 19414 tablet_service.cc:1467] Processing CreateTablet for tablet 547f9b8613fd4f7f949d37c17328f0d4 (DEFAULT_TABLE table=test-table [id=8bb7f5bcf9dc49f58e6bfb205e80daab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:31:54.810010 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.805396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41153 (local address 127.0.0.1:54946)
0927 03:31:54.805631 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:54.805641 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:54.805688 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:54.806101 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:54.806102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:54.806105 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:54.806173 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:54.806175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.807402 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.807403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:54.808467 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:54.808469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.809787 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.809788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:54.809855 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:54.809860 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:31:54.809919 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 03:31:54.809953 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":83,"threads_started":1}
I20260927 03:31:54.810025 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:54.805595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54946 (local address 127.0.98.2:41153)
0927 03:31:54.805800 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:54.805803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:54.805811 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:31:54.805928 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:54.805931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:54.805947 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:54.806033 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:54.806270 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.807319 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.808558 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:54.809709 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:54.809770 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:54.809854 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:54.809934 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 03:31:54.809965 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":97,"threads_started":1}
I20260927 03:31:54.810243 19414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 547f9b8613fd4f7f949d37c17328f0d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:31:54.811808 19152 tablet_service.cc:1467] Processing CreateTablet for tablet 547f9b8613fd4f7f949d37c17328f0d4 (DEFAULT_TABLE table=test-table [id=8bb7f5bcf9dc49f58e6bfb205e80daab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:31:54.812090 19152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 547f9b8613fd4f7f949d37c17328f0d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:31:54.812877 19494 tablet_bootstrap.cc:492] T 547f9b8613fd4f7f949d37c17328f0d4 P 3b3fe43b2125438fb10972ff4cb1dc55: Bootstrap starting.
I20260927 03:31:54.813407 19494 tablet_bootstrap.cc:654] T 547f9b8613fd4f7f949d37c17328f0d4 P 3b3fe43b2125438fb10972ff4cb1dc55: Neither blocks nor log segments found. Creating new log.
I20260927 03:31:54.813701 19494 log.cc:824] T 547f9b8613fd4f7f949d37c17328f0d4 P 3b3fe43b2125438fb10972ff4cb1dc55: Log is configured to *not* fsync() on all Append() calls
I20260927 03:31:54.814347 19494 tablet_bootstrap.cc:492] T 547f9b8613fd4f7f949d37c17328f0d4 P 3b3fe43b2125438fb10972ff4cb1dc55: No bootstrap required, opened a new log
I20260927 03:31:54.814427 19494 ts_tablet_manager.cc:1397] T 547f9b8613fd4f7f949d37c17328f0d4 P 3b3fe43b2125438fb10972ff4cb1dc55: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:31:54.814664 19496 tablet_bootstrap.cc:492] T 547f9b8613fd4f7f949d37c17328f0d4 P f144a169f5494497ba00a3a134fc6ffb: Bootstrap starting.
I20260927 03:31:54.815371 19496 tablet_bootstrap.cc:654] T 547f9b8613fd4f7f949d37c17328f0d4 P f144a169f5494497ba00a3a134fc6ffb: Neither blocks nor log segments found. Creating new log.
I20260927 03:31:54.815454 19283 tablet_service.cc:1467] Processing CreateTablet for tablet 547f9b8613fd4f7f949d37c17328f0d4 (DEFAULT_TABLE table=test-table [id=8bb7f5bcf9dc49f58e6bfb205e80daab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:31:54.815718 19496 log.cc:824] T 547f9b8613fd4f7f949d37c17328f0d4 P f144a169f5494497ba00a3a134fc6ffb: Log is configured to *not* fsync() on all Append() calls
I20260927 03:31:54.815799 19283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 547f9b8613fd4f7f949d37c17328f0d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:31:54.816478 19496 tablet_bootstrap.cc:492] T 547f9b8613fd4f7f949d37c17328f0d4 P f144a169f5494497ba00a3a134fc6ffb: No bootstrap required, opened a new log
I20260927 03:31:54.816545 19496 ts_tablet_manager.cc:1397] T 547f9b8613fd4f7f949d37c17328f0d4 P f144a169f5494497ba00a3a134fc6ffb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:31:54.817263 19494 raft_consensus.cc:348] T 547f9b8613fd4f7f949d37c17328f0d4 P 3b3fe43b2125438fb10972ff4cb1dc55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3fe43b2125438fb10972ff4cb1dc55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35037 } } peers { permanent_uuid: "f144a169f5494497ba00a3a134fc6ffb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44823 } } peers { permanent_uuid: "2da4a91396f54b859ed2e693af576067" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41153 } }
I20260927 03:31:54.817417 19494 raft_consensus.cc:374] T 547f9b8613fd4f7f949d37c17328f0d4 P 3b3fe43b2125438fb10972ff4cb1dc55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:31:54.817461 19494 raft_consensus.cc:729] T 547f9b8613fd4f7f949d37c17328f0d4 P 3b3fe43b2125438fb10972ff4cb1dc55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b3fe43b2125438fb10972ff4cb1dc55, State: Initialized, Role: FOLLOWER
I20260927 03:31:54.817652 19494 consensus_queue.cc:260] T 547f9b8613fd4f7f949d37c17328f0d4 P 3b3fe43b2125438fb10972ff4cb1dc55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3fe43b2125438fb10972ff4cb1dc55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35037 } } peers { permanent_uuid: "f144a169f5494497ba00a3a134fc6ffb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44823 } } peers { permanent_uuid: "2da4a91396f54b859ed2e693af576067" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41153 } }
I20260927 03:31:54.818032 19494 ts_tablet_manager.cc:1428] T 547f9b8613fd4f7f949d37c17328f0d4 P 3b3fe43b2125438fb10972ff4cb1dc55: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 03:31:54.818187 19480 heartbeater.cc:503] Master 127.0.98.62:33871 was elected leader, sending a full tablet report...
I20260927 03:31:54.819108 19496 raft_consensus.cc:348] T 547f9b8613fd4f7f949d37c17328f0d4 P f144a169f5494497ba00a3a134fc6ffb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3fe43b2125438fb10972ff4cb1dc55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35037 } } peers { permanent_uuid: "f144a169f5494497ba00a3a134fc6ffb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44823 } } peers { permanent_uuid: "2da4a91396f54b859ed2e693af576067" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41153 } }
I20260927 03:31:54.819205 19499 tablet_bootstrap.cc:492] T 547f9b8613fd4f7f949d37c17328f0d4 P 2da4a91396f54b859ed2e693af576067: Bootstrap starting.
I20260927 03:31:54.819265 19496 raft_consensus.cc:374] T 547f9b8613fd4f7f949d37c17328f0d4 P f144a169f5494497ba00a3a134fc6ffb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:31:54.819307 19496 raft_consensus.cc:729] T 547f9b8613fd4f7f949d37c17328f0d4 P f144a169f5494497ba00a3a134fc6ffb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f144a169f5494497ba00a3a134fc6ffb, State: Initialized, Role: FOLLOWER
I20260927 03:31:54.819449 19496 consensus_queue.cc:260] T 547f9b8613fd4f7f949d37c17328f0d4 P f144a169f5494497ba00a3a134fc6ffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3fe43b2125438fb10972ff4cb1dc55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35037 } } peers { permanent_uuid: "f144a169f5494497ba00a3a134fc6ffb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44823 } } peers { permanent_uuid: "2da4a91396f54b859ed2e693af576067" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41153 } }
I20260927 03:31:54.819720 19496 ts_tablet_manager.cc:1428] T 547f9b8613fd4f7f949d37c17328f0d4 P f144a169f5494497ba00a3a134fc6ffb: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260927 03:31:54.819767 19218 heartbeater.cc:503] Master 127.0.98.62:33871 was elected leader, sending a full tablet report...
I20260927 03:31:54.819868 19499 tablet_bootstrap.cc:654] T 547f9b8613fd4f7f949d37c17328f0d4 P 2da4a91396f54b859ed2e693af576067: Neither blocks nor log segments found. Creating new log.
I20260927 03:31:54.820178 19499 log.cc:824] T 547f9b8613fd4f7f949d37c17328f0d4 P 2da4a91396f54b859ed2e693af576067: Log is configured to *not* fsync() on all Append() calls
I20260927 03:31:54.820827 19499 tablet_bootstrap.cc:492] T 547f9b8613fd4f7f949d37c17328f0d4 P 2da4a91396f54b859ed2e693af576067: No bootstrap required, opened a new log
I20260927 03:31:54.820905 19499 ts_tablet_manager.cc:1397] T 547f9b8613fd4f7f949d37c17328f0d4 P 2da4a91396f54b859ed2e693af576067: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:31:54.822659 19499 raft_consensus.cc:348] T 547f9b8613fd4f7f949d37c17328f0d4 P 2da4a91396f54b859ed2e693af576067 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3fe43b2125438fb10972ff4cb1dc55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35037 } } peers { permanent_uuid: "f144a169f5494497ba00a3a134fc6ffb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44823 } } peers { permanent_uuid: "2da4a91396f54b859ed2e693af576067" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41153 } }
I20260927 03:31:54.822801 19499 raft_consensus.cc:374] T 547f9b8613fd4f7f949d37c17328f0d4 P 2da4a91396f54b859ed2e693af576067 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:31:54.822846 19499 raft_consensus.cc:729] T 547f9b8613fd4f7f949d37c17328f0d4 P 2da4a91396f54b859ed2e693af576067 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2da4a91396f54b859ed2e693af576067, State: Initialized, Role: FOLLOWER
I20260927 03:31:54.822978 19499 consensus_queue.cc:260] T 547f9b8613fd4f7f949d37c17328f0d4 P 2da4a91396f54b859ed2e693af576067 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3fe43b2125438fb10972ff4cb1dc55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35037 } } peers { permanent_uuid: "f144a169f5494497ba00a3a134fc6ffb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44823 } } peers { permanent_uuid: "2da4a91396f54b859ed2e693af576067" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41153 } }
I20260927 03:31:54.823207 19499 ts_tablet_manager.cc:1428] T 547f9b8613fd4f7f949d37c17328f0d4 P 2da4a91396f54b859ed2e693af576067: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 03:31:54.823282 19349 heartbeater.cc:503] Master 127.0.98.62:33871 was elected leader, sending a full tablet report...
W20260927 03:31:54.909513 19350 tablet.cc:2369] T 547f9b8613fd4f7f949d37c17328f0d4 P 2da4a91396f54b859ed2e693af576067: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:31:55.014000 19219 tablet.cc:2369] T 547f9b8613fd4f7f949d37c17328f0d4 P f144a169f5494497ba00a3a134fc6ffb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:31:55.037968 19481 tablet.cc:2369] T 547f9b8613fd4f7f949d37c17328f0d4 P 3b3fe43b2125438fb10972ff4cb1dc55: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:31:55.062361 19500 raft_consensus.cc:482] T 547f9b8613fd4f7f949d37c17328f0d4 P f144a169f5494497ba00a3a134fc6ffb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:31:55.062505 19500 raft_consensus.cc:504] T 547f9b8613fd4f7f949d37c17328f0d4 P f144a169f5494497ba00a3a134fc6ffb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3fe43b2125438fb10972ff4cb1dc55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35037 } } peers { permanent_uuid: "f144a169f5494497ba00a3a134fc6ffb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44823 } } peers { permanent_uuid: "2da4a91396f54b859ed2e693af576067" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41153 } }
I20260927 03:31:55.062831 19500 leader_election.cc:288] T 547f9b8613fd4f7f949d37c17328f0d4 P f144a169f5494497ba00a3a134fc6ffb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b3fe43b2125438fb10972ff4cb1dc55 (127.0.98.3:35037), 2da4a91396f54b859ed2e693af576067 (127.0.98.2:41153)
I20260927 03:31:55.065793 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:55.063134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52205 (local address 127.0.98.3:35037)
0927 03:31:55.063192 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:55.063196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:55.063229 (+ 33us) server_negotiation.cc:407] Connection header received
0927 03:31:55.063300 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:55.063301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:55.063311 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:55.063388 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:55.063593 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:55.064062 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:55.065063 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:55.065448 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:55.065538 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:55.065548 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:55.065691 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 03:31:55.065729 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 03:31:55.065814 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:55.063035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35037 (local address 127.0.98.1:52205)
0927 03:31:55.063163 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:55.063176 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:55.063232 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:55.063395 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:55.063396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:55.063399 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:55.063508 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:55.063511 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:55.064151 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:55.064152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:55.064962 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:55.064964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:55.065539 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:55.065540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:55.065615 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:55.065619 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:31:55.065685 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 03:31:55.065723 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":8,"thread_start_us":79,"threads_started":1}
I20260927 03:31:55.065937 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:55.063313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60243 (local address 127.0.98.2:41153)
0927 03:31:55.063368 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:55.063370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:55.063380 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:31:55.063459 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:55.063460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:55.063484 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:55.063558 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:55.063716 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:55.064179 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:55.065168 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:55.065562 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:55.065653 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:55.065664 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:55.065810 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 03:31:55.065847 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 03:31:55.066147 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:55.063227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41153 (local address 127.0.98.1:60243)
0927 03:31:55.063336 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:55.063344 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:55.063382 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:55.063562 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:55.063563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:55.063564 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:55.063641 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:55.063643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:55.064267 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:55.064268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:55.065063 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:55.065065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:55.065666 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:55.065667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:55.065735 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:55.065738 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:31:55.065805 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 03:31:55.065843 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":39,"threads_started":1}
I20260927 03:31:55.066325 19434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "547f9b8613fd4f7f949d37c17328f0d4" candidate_uuid: "f144a169f5494497ba00a3a134fc6ffb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b3fe43b2125438fb10972ff4cb1dc55" is_pre_election: true
I20260927 03:31:55.066535 19434 raft_consensus.cc:2454] T 547f9b8613fd4f7f949d37c17328f0d4 P 3b3fe43b2125438fb10972ff4cb1dc55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f144a169f5494497ba00a3a134fc6ffb in term 0.
I20260927 03:31:55.066517 19303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "547f9b8613fd4f7f949d37c17328f0d4" candidate_uuid: "f144a169f5494497ba00a3a134fc6ffb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2da4a91396f54b859ed2e693af576067" is_pre_election: true
I20260927 03:31:55.066684 19303 raft_consensus.cc:2454] T 547f9b8613fd4f7f949d37c17328f0d4 P 2da4a91396f54b859ed2e693af576067 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f144a169f5494497ba00a3a134fc6ffb in term 0.
I20260927 03:31:55.066789 19108 leader_election.cc:302] T 547f9b8613fd4f7f949d37c17328f0d4 P f144a169f5494497ba00a3a134fc6ffb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b3fe43b2125438fb10972ff4cb1dc55, f144a169f5494497ba00a3a134fc6ffb; no voters:
I20260927 03:31:55.066917 19500 raft_consensus.cc:2781] T 547f9b8613fd4f7f949d37c17328f0d4 P f144a169f5494497ba00a3a134fc6ffb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:31:55.066979 19500 raft_consensus.cc:482] T 547f9b8613fd4f7f949d37c17328f0d4 P f144a169f5494497ba00a3a134fc6ffb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:31:55.066995 19500 raft_consensus.cc:3037] T 547f9b8613fd4f7f949d37c17328f0d4 P f144a169f5494497ba00a3a134fc6ffb [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:31:55.067746 19500 raft_consensus.cc:504] T 547f9b8613fd4f7f949d37c17328f0d4 P f144a169f5494497ba00a3a134fc6ffb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3fe43b2125438fb10972ff4cb1dc55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35037 } } peers { permanent_uuid: "f144a169f5494497ba00a3a134fc6ffb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44823 } } peers { permanent_uuid: "2da4a91396f54b859ed2e693af576067" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41153 } }
I20260927 03:31:55.067896 19500 leader_election.cc:288] T 547f9b8613fd4f7f949d37c17328f0d4 P f144a169f5494497ba00a3a134fc6ffb [CANDIDATE]: Term 1 election: Requested vote from peers 3b3fe43b2125438fb10972ff4cb1dc55 (127.0.98.3:35037), 2da4a91396f54b859ed2e693af576067 (127.0.98.2:41153)
I20260927 03:31:55.068070 19434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "547f9b8613fd4f7f949d37c17328f0d4" candidate_uuid: "f144a169f5494497ba00a3a134fc6ffb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b3fe43b2125438fb10972ff4cb1dc55"
I20260927 03:31:55.068107 19303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "547f9b8613fd4f7f949d37c17328f0d4" candidate_uuid: "f144a169f5494497ba00a3a134fc6ffb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2da4a91396f54b859ed2e693af576067"
I20260927 03:31:55.068177 19303 raft_consensus.cc:3037] T 547f9b8613fd4f7f949d37c17328f0d4 P 2da4a91396f54b859ed2e693af576067 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:31:55.068185 19434 raft_consensus.cc:3037] T 547f9b8613fd4f7f949d37c17328f0d4 P 3b3fe43b2125438fb10972ff4cb1dc55 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:31:55.068965 19303 raft_consensus.cc:2454] T 547f9b8613fd4f7f949d37c17328f0d4 P 2da4a91396f54b859ed2e693af576067 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f144a169f5494497ba00a3a134fc6ffb in term 1.
I20260927 03:31:55.068984 19434 raft_consensus.cc:2454] T 547f9b8613fd4f7f949d37c17328f0d4 P 3b3fe43b2125438fb10972ff4cb1dc55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f144a169f5494497ba00a3a134fc6ffb in term 1.
I20260927 03:31:55.069141 19108 leader_election.cc:302] T 547f9b8613fd4f7f949d37c17328f0d4 P f144a169f5494497ba00a3a134fc6ffb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2da4a91396f54b859ed2e693af576067, f144a169f5494497ba00a3a134fc6ffb; no voters:
I20260927 03:31:55.069226 19500 raft_consensus.cc:2781] T 547f9b8613fd4f7f949d37c17328f0d4 P f144a169f5494497ba00a3a134fc6ffb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:31:55.069378 19500 raft_consensus.cc:686] T 547f9b8613fd4f7f949d37c17328f0d4 P f144a169f5494497ba00a3a134fc6ffb [term 1 LEADER]: Becoming Leader. State: Replica: f144a169f5494497ba00a3a134fc6ffb, State: Running, Role: LEADER
I20260927 03:31:55.069476 19500 consensus_queue.cc:237] T 547f9b8613fd4f7f949d37c17328f0d4 P f144a169f5494497ba00a3a134fc6ffb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3fe43b2125438fb10972ff4cb1dc55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35037 } } peers { permanent_uuid: "f144a169f5494497ba00a3a134fc6ffb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44823 } } peers { permanent_uuid: "2da4a91396f54b859ed2e693af576067" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41153 } }
I20260927 03:31:55.070271 19030 catalog_manager.cc:5495] T 547f9b8613fd4f7f949d37c17328f0d4 P f144a169f5494497ba00a3a134fc6ffb reported cstate change: term changed from 0 to 1, leader changed from <none> to f144a169f5494497ba00a3a134fc6ffb (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f144a169f5494497ba00a3a134fc6ffb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b3fe43b2125438fb10972ff4cb1dc55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f144a169f5494497ba00a3a134fc6ffb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2da4a91396f54b859ed2e693af576067" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41153 } health_report { overall_health: UNKNOWN } } }
I20260927 03:31:55.097188 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:55.094599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43222 (local address 127.0.98.1:44823)
0927 03:31:55.094644 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:55.094646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:55.094654 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:31:55.094702 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:55.094703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:55.094735 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:55.094824 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:31:55.094979 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:55.095556 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:55.096396 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:55.096607 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:55.096686 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:55.096854 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:31:55.096952 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:31:55.097064 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:55.097095 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 03:31:55.097105 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:31:55.099037 19434 raft_consensus.cc:1261] T 547f9b8613fd4f7f949d37c17328f0d4 P 3b3fe43b2125438fb10972ff4cb1dc55 [term 1 FOLLOWER]: Refusing update from remote peer f144a169f5494497ba00a3a134fc6ffb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:31:55.099094 19303 raft_consensus.cc:1261] T 547f9b8613fd4f7f949d37c17328f0d4 P 2da4a91396f54b859ed2e693af576067 [term 1 FOLLOWER]: Refusing update from remote peer f144a169f5494497ba00a3a134fc6ffb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:31:55.099406 19505 consensus_queue.cc:1035] T 547f9b8613fd4f7f949d37c17328f0d4 P f144a169f5494497ba00a3a134fc6ffb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2da4a91396f54b859ed2e693af576067" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:31:55.099606 19500 consensus_queue.cc:1035] T 547f9b8613fd4f7f949d37c17328f0d4 P f144a169f5494497ba00a3a134fc6ffb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b3fe43b2125438fb10972ff4cb1dc55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:31:55.102178 19508 mvcc.cc:205] Tried to move back new op lower bound from 7333805732242784256 to 7333805732125245440. Current Snapshot: MvccSnapshot[applied={T|T < 7333805732242784256}]
I20260927 03:31:55.103075 19510 mvcc.cc:205] Tried to move back new op lower bound from 7333805732242784256 to 7333805732125245440. Current Snapshot: MvccSnapshot[applied={T|T < 7333805732242784256}]
I20260927 03:31:55.103248 19509 mvcc.cc:205] Tried to move back new op lower bound from 7333805732242784256 to 7333805732125245440. Current Snapshot: MvccSnapshot[applied={T|T < 7333805732242784256 or (T in {7333805732242784256})}]
I20260927 03:31:55.104534 19029 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:31:55.104652 19029 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:31:55.105417 19029 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d08b17bab784b7680c142b24a4c133d: Sending DeleteTablet for 3 replicas of tablet 547f9b8613fd4f7f949d37c17328f0d4
I20260927 03:31:55.105785 19414 tablet_service.cc:1514] Processing DeleteTablet for tablet 547f9b8613fd4f7f949d37c17328f0d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:31:55 UTC) from {username='slave'} at 127.0.0.1:56478
I20260927 03:31:55.105826 19152 tablet_service.cc:1514] Processing DeleteTablet for tablet 547f9b8613fd4f7f949d37c17328f0d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:31:55 UTC) from {username='slave'} at 127.0.0.1:43210
I20260927 03:31:55.106045 19283 tablet_service.cc:1514] Processing DeleteTablet for tablet 547f9b8613fd4f7f949d37c17328f0d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:31:55 UTC) from {username='slave'} at 127.0.0.1:54946
I20260927 03:31:55.106055 19517 tablet_replica.cc:330] T 547f9b8613fd4f7f949d37c17328f0d4 P 3b3fe43b2125438fb10972ff4cb1dc55: stopping tablet replica
I20260927 03:31:55.106165 19517 raft_consensus.cc:2229] T 547f9b8613fd4f7f949d37c17328f0d4 P 3b3fe43b2125438fb10972ff4cb1dc55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:31:55.106268 19517 raft_consensus.cc:2258] T 547f9b8613fd4f7f949d37c17328f0d4 P 3b3fe43b2125438fb10972ff4cb1dc55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:31:55.106287 19519 tablet_replica.cc:330] T 547f9b8613fd4f7f949d37c17328f0d4 P 2da4a91396f54b859ed2e693af576067: stopping tablet replica
I20260927 03:31:55.106427 19518 tablet_replica.cc:330] T 547f9b8613fd4f7f949d37c17328f0d4 P f144a169f5494497ba00a3a134fc6ffb: stopping tablet replica
I20260927 03:31:55.106514 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 19090
I20260927 03:31:55.106524 19518 raft_consensus.cc:2229] T 547f9b8613fd4f7f949d37c17328f0d4 P f144a169f5494497ba00a3a134fc6ffb [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:31:55.106545 19519 raft_consensus.cc:2229] T 547f9b8613fd4f7f949d37c17328f0d4 P 2da4a91396f54b859ed2e693af576067 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:31:55.106619 19518 raft_consensus.cc:2258] T 547f9b8613fd4f7f949d37c17328f0d4 P f144a169f5494497ba00a3a134fc6ffb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:31:55.108255 19519 raft_consensus.cc:2258] T 547f9b8613fd4f7f949d37c17328f0d4 P 2da4a91396f54b859ed2e693af576067 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:31:55.108644 19519 ts_tablet_manager.cc:1905] T 547f9b8613fd4f7f949d37c17328f0d4 P 2da4a91396f54b859ed2e693af576067: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:31:55.110369 19519 ts_tablet_manager.cc:1918] T 547f9b8613fd4f7f949d37c17328f0d4 P 2da4a91396f54b859ed2e693af576067: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:31:55.110451 19519 log.cc:1196] T 547f9b8613fd4f7f949d37c17328f0d4 P 2da4a91396f54b859ed2e693af576067: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-1/wal/wals/547f9b8613fd4f7f949d37c17328f0d4
I20260927 03:31:55.110720 19519 ts_tablet_manager.cc:1939] T 547f9b8613fd4f7f949d37c17328f0d4 P 2da4a91396f54b859ed2e693af576067: Deleting consensus metadata
I20260927 03:31:55.111130 19017 catalog_manager.cc:4841] TS 2da4a91396f54b859ed2e693af576067 (127.0.98.2:41153): tablet 547f9b8613fd4f7f949d37c17328f0d4 (table test-table [id=8bb7f5bcf9dc49f58e6bfb205e80daab]) successfully deleted
W20260927 03:31:55.111491 19016 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44823 (error 108)
I20260927 03:31:55.111795 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 19221
I20260927 03:31:55.112329 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:55.111889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44823 (local address 127.0.0.1:43230)
0927 03:31:55.112249 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:55.112289 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341}
W20260927 03:31:55.112468 19016 catalog_manager.cc:4584] TS f144a169f5494497ba00a3a134fc6ffb (127.0.98.1:44823): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 547f9b8613fd4f7f949d37c17328f0d4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44823: connect: Connection refused (error 111)
I20260927 03:31:55.112679 19517 ts_tablet_manager.cc:1905] T 547f9b8613fd4f7f949d37c17328f0d4 P 3b3fe43b2125438fb10972ff4cb1dc55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:31:55.114048 19517 ts_tablet_manager.cc:1918] T 547f9b8613fd4f7f949d37c17328f0d4 P 3b3fe43b2125438fb10972ff4cb1dc55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:31:55.114137 19517 log.cc:1196] T 547f9b8613fd4f7f949d37c17328f0d4 P 3b3fe43b2125438fb10972ff4cb1dc55: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790479675865103-392-0/minicluster-data/ts-2/wal/wals/547f9b8613fd4f7f949d37c17328f0d4
I20260927 03:31:55.114388 19517 ts_tablet_manager.cc:1939] T 547f9b8613fd4f7f949d37c17328f0d4 P 3b3fe43b2125438fb10972ff4cb1dc55: Deleting consensus metadata
I20260927 03:31:55.114835 19017 catalog_manager.cc:4841] TS 3b3fe43b2125438fb10972ff4cb1dc55 (127.0.98.3:35037): tablet 547f9b8613fd4f7f949d37c17328f0d4 (table test-table [id=8bb7f5bcf9dc49f58e6bfb205e80daab]) successfully deleted
I20260927 03:31:55.117483 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 19352
I20260927 03:31:55.122099 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 18999
2026-09-27T03:31:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (906 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:31:55 dist-test-slave-j0hf krb5kdc[19523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:31:55 dist-test-slave-j0hf krb5kdc[19523](info): set up 2 sockets
Sep 27 03:31:55 dist-test-slave-j0hf krb5kdc[19523](info): commencing operation
krb5kdc: starting...
W20260927 03:31:56.171574 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:31:56 dist-test-slave-j0hf krb5kdc[19523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:31:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:31:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:31:56.288317 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46509
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:31:56.370074 19536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:56.370283 19536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:56.370304 19536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:56.371384 19536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:31:56.371476 19536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:56.371519 19536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:56.371534 19536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:31:56.371546 19536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:31:56.372970 19536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:56.373157 19536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:56.373324 19536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:31:56.377890 19541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:56.377902 19542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:56.377890 19544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:56.378228 19536 server_base.cc:1034] running on GCE node
I20260927 03:31:56.378497 19536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:56.378814 19536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:56.379949 19536 hybrid_clock.cc:648] HybridClock initialized: now 1790479916379920 us; error 41 us; skew 500 ppm
Sep 27 03:31:56 dist-test-slave-j0hf krb5kdc[19523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:31:56.382459 19536 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:31:56.382846 19536 webserver.cc:460] Webserver started at http://127.0.98.62:33997/ using document root <none> and password file <none>
I20260927 03:31:56.383024 19536 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:56.383064 19536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:56.383147 19536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:31:56.384188 19536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "e576a32023e7459e8d84b6834eb559c0"
format_stamp: "Formatted at 2026-09-27 03:31:56 on dist-test-slave-j0hf"
server_key: "56218b92de1cc656919357ba9001ca45"
server_key_iv: "62ccba0ab5dea4557ca12a634792c7a0"
server_key_version: "encryptionkey@0"
I20260927 03:31:56.384481 19536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "e576a32023e7459e8d84b6834eb559c0"
format_stamp: "Formatted at 2026-09-27 03:31:56 on dist-test-slave-j0hf"
server_key: "56218b92de1cc656919357ba9001ca45"
server_key_iv: "62ccba0ab5dea4557ca12a634792c7a0"
server_key_version: "encryptionkey@0"
I20260927 03:31:56.385704 19536 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:31:56.386375 19551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:56.386543 19536 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:31:56.386642 19536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "e576a32023e7459e8d84b6834eb559c0"
format_stamp: "Formatted at 2026-09-27 03:31:56 on dist-test-slave-j0hf"
server_key: "56218b92de1cc656919357ba9001ca45"
server_key_iv: "62ccba0ab5dea4557ca12a634792c7a0"
server_key_version: "encryptionkey@0"
I20260927 03:31:56.386715 19536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:56.398561 19536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:56.400457 19536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:56.400620 19536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:56.404670 19536 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46509
I20260927 03:31:56.404721 19603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46509 every 8 connection(s)
I20260927 03:31:56.405097 19536 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:31:56.405958 19604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:31:56.408298 19604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e576a32023e7459e8d84b6834eb559c0: Bootstrap starting.
I20260927 03:31:56.408885 19604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e576a32023e7459e8d84b6834eb559c0: Neither blocks nor log segments found. Creating new log.
I20260927 03:31:56.409176 19604 log.cc:824] T 00000000000000000000000000000000 P e576a32023e7459e8d84b6834eb559c0: Log is configured to *not* fsync() on all Append() calls
I20260927 03:31:56.409869 19604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e576a32023e7459e8d84b6834eb559c0: No bootstrap required, opened a new log
I20260927 03:31:56.411509 19604 raft_consensus.cc:348] T 00000000000000000000000000000000 P e576a32023e7459e8d84b6834eb559c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e576a32023e7459e8d84b6834eb559c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46509 } }
I20260927 03:31:56.411656 19604 raft_consensus.cc:374] T 00000000000000000000000000000000 P e576a32023e7459e8d84b6834eb559c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:31:56.411679 19604 raft_consensus.cc:729] T 00000000000000000000000000000000 P e576a32023e7459e8d84b6834eb559c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e576a32023e7459e8d84b6834eb559c0, State: Initialized, Role: FOLLOWER
I20260927 03:31:56.411855 19604 consensus_queue.cc:260] T 00000000000000000000000000000000 P e576a32023e7459e8d84b6834eb559c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e576a32023e7459e8d84b6834eb559c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46509 } }
I20260927 03:31:56.411937 19604 raft_consensus.cc:388] T 00000000000000000000000000000000 P e576a32023e7459e8d84b6834eb559c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:31:56.411989 19604 raft_consensus.cc:482] T 00000000000000000000000000000000 P e576a32023e7459e8d84b6834eb559c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:31:56.412039 19604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e576a32023e7459e8d84b6834eb559c0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:31:56.412637 19604 raft_consensus.cc:504] T 00000000000000000000000000000000 P e576a32023e7459e8d84b6834eb559c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e576a32023e7459e8d84b6834eb559c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46509 } }
I20260927 03:31:56.412730 19604 leader_election.cc:302] T 00000000000000000000000000000000 P e576a32023e7459e8d84b6834eb559c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e576a32023e7459e8d84b6834eb559c0; no voters:
I20260927 03:31:56.412886 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 19536
I20260927 03:31:56.412961 19604 leader_election.cc:288] T 00000000000000000000000000000000 P e576a32023e7459e8d84b6834eb559c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:31:56.412993 19607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e576a32023e7459e8d84b6834eb559c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:31:56.413038 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:31:56.413157 19607 raft_consensus.cc:686] T 00000000000000000000000000000000 P e576a32023e7459e8d84b6834eb559c0 [term 1 LEADER]: Becoming Leader. State: Replica: e576a32023e7459e8d84b6834eb559c0, State: Running, Role: LEADER
I20260927 03:31:56.413267 392 external_mini_cluster.cc:1428] Setting key 7c0ba1b8f436ec7cbbb97d90ba2be06f
I20260927 03:31:56.413312 19604 sys_catalog.cc:564] T 00000000000000000000000000000000 P e576a32023e7459e8d84b6834eb559c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:31:56.413319 19607 consensus_queue.cc:237] T 00000000000000000000000000000000 P e576a32023e7459e8d84b6834eb559c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e576a32023e7459e8d84b6834eb559c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46509 } }
I20260927 03:31:56.413864 19609 sys_catalog.cc:455] T 00000000000000000000000000000000 P e576a32023e7459e8d84b6834eb559c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e576a32023e7459e8d84b6834eb559c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e576a32023e7459e8d84b6834eb559c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46509 } } }
I20260927 03:31:56.414165 19609 sys_catalog.cc:458] T 00000000000000000000000000000000 P e576a32023e7459e8d84b6834eb559c0 [sys.catalog]: This master's current role is: LEADER
I20260927 03:31:56.414090 19608 sys_catalog.cc:455] T 00000000000000000000000000000000 P e576a32023e7459e8d84b6834eb559c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e576a32023e7459e8d84b6834eb559c0. Latest consensus state: current_term: 1 leader_uuid: "e576a32023e7459e8d84b6834eb559c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e576a32023e7459e8d84b6834eb559c0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46509 } } }
I20260927 03:31:56.414444 19608 sys_catalog.cc:458] T 00000000000000000000000000000000 P e576a32023e7459e8d84b6834eb559c0 [sys.catalog]: This master's current role is: LEADER
I20260927 03:31:56.414636 19619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:31:56.415475 19619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:31:56.419853 19619 catalog_manager.cc:1347] Generated new cluster ID: 932125d1d7e0426f9fda6beee9a6ba39
I20260927 03:31:56.419929 19619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:31:56 dist-test-slave-j0hf krb5kdc[19523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:31:56.424932 19616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:56.413894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37112 (local address 127.0.98.62:46509)
0927 03:31:56.414074 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:56.414085 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:56.414149 (+ 64us) server_negotiation.cc:407] Connection header received
0927 03:31:56.414322 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:56.414331 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:56.414403 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:56.415152 (+ 749us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:56.415390 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:56.416654 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:56.417454 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:56.417891 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:56.418018 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:56.420651 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:56.420654 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:56.420655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:31:56.420666 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:56.423238 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:56.423684 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:56.423688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:56.423688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:56.423803 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:56.424010 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:56.424011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:56.424011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:56.424307 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:56.424455 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:56.424573 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 03:31:56.424704 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":73,"threads_started":1}
I20260927 03:31:56.442232 19619 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:31:56.442785 19619 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:31:56.445086 19619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e576a32023e7459e8d84b6834eb559c0: Generated new TSK 0
I20260927 03:31:56.445286 19619 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 03:31:56.474655 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46509
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:31:56.556035 19630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:56.556269 19630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:56.556293 19630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:56.557400 19630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:56.557435 19630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:56.557461 19630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:31:56.558873 19630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:56.559103 19630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:56.559279 19630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:31:56.564139 19636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:56.564170 19638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:56.564206 19635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:56.564379 19630 server_base.cc:1034] running on GCE node
I20260927 03:31:56.564594 19630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:56.564782 19630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:56.565937 19630 hybrid_clock.cc:648] HybridClock initialized: now 1790479916565927 us; error 38 us; skew 500 ppm
Sep 27 03:31:56 dist-test-slave-j0hf krb5kdc[19523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:31:56.568509 19630 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 03:31:56.568934 19630 webserver.cc:460] Webserver started at http://127.0.98.1:35269/ using document root <none> and password file <none>
I20260927 03:31:56.569151 19630 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:56.569216 19630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:56.569365 19630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:31:56.570623 19630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-0/data/instance:
uuid: "ffe1b22521304b9eb8f053784d407b9d"
format_stamp: "Formatted at 2026-09-27 03:31:56 on dist-test-slave-j0hf"
server_key: "1d5fe20cd3ed92c754baf0f6aa306a60"
server_key_iv: "35666991322de0e1d49031af07ff7cff"
server_key_version: "encryptionkey@0"
I20260927 03:31:56.571009 19630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ffe1b22521304b9eb8f053784d407b9d"
format_stamp: "Formatted at 2026-09-27 03:31:56 on dist-test-slave-j0hf"
server_key: "1d5fe20cd3ed92c754baf0f6aa306a60"
server_key_iv: "35666991322de0e1d49031af07ff7cff"
server_key_version: "encryptionkey@0"
I20260927 03:31:56.572381 19630 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:31:56.573078 19645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:56.573268 19630 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:31:56.573359 19630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "ffe1b22521304b9eb8f053784d407b9d"
format_stamp: "Formatted at 2026-09-27 03:31:56 on dist-test-slave-j0hf"
server_key: "1d5fe20cd3ed92c754baf0f6aa306a60"
server_key_iv: "35666991322de0e1d49031af07ff7cff"
server_key_version: "encryptionkey@0"
I20260927 03:31:56.573426 19630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:56.585065 19630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:56.587368 19630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:56.587585 19630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:56.587826 19630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:31:56.588117 19630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:31:56.588150 19630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:56.588171 19630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:31:56.588212 19630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:56.593979 19630 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40709
I20260927 03:31:56.594061 19758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40709 every 8 connection(s)
I20260927 03:31:56.594357 19630 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:31:56.599584 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 19630
I20260927 03:31:56.599702 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:31:56.599929 392 external_mini_cluster.cc:1428] Setting key 3775c826f9c7b8ed7e90dadc801a404a
Sep 27 03:31:56 dist-test-slave-j0hf krb5kdc[19523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:31:56.604545 19616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:56.595074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42877 (local address 127.0.98.62:46509)
0927 03:31:56.595126 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:56.595128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:56.595371 (+ 243us) server_negotiation.cc:407] Connection header received
0927 03:31:56.596361 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:56.596362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:56.596376 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:56.596432 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:56.597036 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:56.597534 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:56.598606 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:56.598791 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:56.598839 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:56.601411 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:56.601413 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:56.601413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:31:56.601423 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:56.603115 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:56.603589 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:56.603590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:56.603591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:56.603651 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:56.603900 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:56.603901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:56.603902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:56.604022 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:56.604089 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:56.604370 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 03:31:56.604456 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:31:56.604746 19761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:56.595084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46509 (local address 127.0.98.1:42877)
0927 03:31:56.595248 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:56.595283 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:56.596263 (+ 980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:56.596504 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:56.596507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:56.596756 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:31:56.596948 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:56.596952 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:56.597697 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:56.597697 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:56.598530 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:56.598533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:56.598857 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:56.598858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:56.598955 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:56.599554 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:31:56.599563 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:31:56.601303 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:31:56.603210 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:31:56.603217 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:31:56.603220 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:31:56.603501 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:31:56.603714 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:31:56.603715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:31:56.603715 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:31:56.603830 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:31:56.604095 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:31:56.604102 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:31:56.604239 (+ 137us) client_negotiation.cc:769] Sending connection context
0927 03:31:56.604336 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 03:31:56.604505 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
I20260927 03:31:56.605247 19759 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46509
I20260927 03:31:56.605340 19759 heartbeater.cc:464] Registering TS with master...
I20260927 03:31:56.605535 19759 heartbeater.cc:511] Master 127.0.98.62:46509 requested a full tablet report, sending...
I20260927 03:31:56.606103 19568 ts_manager.cc:194] Registered new tserver with Master: ffe1b22521304b9eb8f053784d407b9d (127.0.98.1:40709)
I20260927 03:31:56.607055 19568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42877
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 03:31:56.626080 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46509
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:31:56.707403 19764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:56.707638 19764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:56.707659 19764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:56.708760 19764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:56.708791 19764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:56.708817 19764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:31:56.710191 19764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:56.710430 19764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:56.710629 19764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:31:56.716009 19770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:56.715994 19769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:56.716063 19764 server_base.cc:1034] running on GCE node
W20260927 03:31:56.715983 19772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:56.716487 19764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:56.716694 19764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:56.717836 19764 hybrid_clock.cc:648] HybridClock initialized: now 1790479916717805 us; error 46 us; skew 500 ppm
Sep 27 03:31:56 dist-test-slave-j0hf krb5kdc[19523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:31:56.720413 19764 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 03:31:56.720860 19764 webserver.cc:460] Webserver started at http://127.0.98.2:38161/ using document root <none> and password file <none>
I20260927 03:31:56.721096 19764 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:56.721165 19764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:56.721316 19764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:31:56.722394 19764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-1/data/instance:
uuid: "02f1ff87e47c4d59918edb01ec8dd82a"
format_stamp: "Formatted at 2026-09-27 03:31:56 on dist-test-slave-j0hf"
server_key: "fef73fbf61b59034e7364115fb6c4c69"
server_key_iv: "521deb81b61ab55ee475615d4f9c5d69"
server_key_version: "encryptionkey@0"
I20260927 03:31:56.722759 19764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-1/wal/instance:
uuid: "02f1ff87e47c4d59918edb01ec8dd82a"
format_stamp: "Formatted at 2026-09-27 03:31:56 on dist-test-slave-j0hf"
server_key: "fef73fbf61b59034e7364115fb6c4c69"
server_key_iv: "521deb81b61ab55ee475615d4f9c5d69"
server_key_version: "encryptionkey@0"
I20260927 03:31:56.724223 19764 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:31:56.724934 19779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:56.725104 19764 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:56.725207 19764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "02f1ff87e47c4d59918edb01ec8dd82a"
format_stamp: "Formatted at 2026-09-27 03:31:56 on dist-test-slave-j0hf"
server_key: "fef73fbf61b59034e7364115fb6c4c69"
server_key_iv: "521deb81b61ab55ee475615d4f9c5d69"
server_key_version: "encryptionkey@0"
I20260927 03:31:56.725284 19764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:56.753302 19764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:56.755329 19764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:56.755541 19764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:56.755820 19764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:31:56.756163 19764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:31:56.756222 19764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:56.756263 19764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:31:56.756310 19764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:56.762697 19764 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45135
I20260927 03:31:56.762800 19892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45135 every 8 connection(s)
I20260927 03:31:56.763130 19764 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:31:56 dist-test-slave-j0hf krb5kdc[19523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:31:56.771238 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 19764
I20260927 03:31:56.771370 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:31:56.771641 392 external_mini_cluster.cc:1428] Setting key d4dd15954b9fba1ecd1c6b3fd1466643
I20260927 03:31:56.772459 19616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:56.763923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48507 (local address 127.0.98.62:46509)
0927 03:31:56.764004 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:56.764006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:56.764213 (+ 207us) server_negotiation.cc:407] Connection header received
0927 03:31:56.765150 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:56.765151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:56.765161 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:56.765220 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:56.765830 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:56.766323 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:56.767354 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:56.767567 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:56.767617 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:56.769675 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:56.769677 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:56.769677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:31:56.769684 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:56.770870 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:56.771363 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:56.771364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:56.771365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:56.771465 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:56.771774 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:56.771775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:56.771775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:56.771879 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:56.771967 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:56.772289 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 03:31:56.772364 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 03:31:56.772706 19895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:56.763911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46509 (local address 127.0.98.2:48507)
0927 03:31:56.764081 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:56.764124 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:56.765055 (+ 931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:56.765286 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:56.765289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:56.765555 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:31:56.765751 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:56.765758 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:56.766400 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:56.766401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:56.767276 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:56.767278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:56.767636 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:56.767636 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:56.767743 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:56.768209 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:31:56.768217 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:31:56.769586 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:31:56.770942 (+ 1356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:31:56.770948 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:31:56.770950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:31:56.771282 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:31:56.771527 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:31:56.771528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:31:56.771529 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:31:56.771670 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:31:56.771960 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:31:56.771965 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:31:56.772137 (+ 172us) client_negotiation.cc:769] Sending connection context
0927 03:31:56.772261 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 03:31:56.772449 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260927 03:31:56.773185 19893 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46509
I20260927 03:31:56.773288 19893 heartbeater.cc:464] Registering TS with master...
I20260927 03:31:56.773463 19893 heartbeater.cc:511] Master 127.0.98.62:46509 requested a full tablet report, sending...
I20260927 03:31:56.773847 19568 ts_manager.cc:194] Registered new tserver with Master: 02f1ff87e47c4d59918edb01ec8dd82a (127.0.98.2:45135)
I20260927 03:31:56.774220 19568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48507
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 03:31:56.796352 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46509
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:31:56.875437 19898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:56.875639 19898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:56.875718 19898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:56.877051 19898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:56.877099 19898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:56.877126 19898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:31:56.878575 19898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:56.878769 19898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:56.878942 19898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:31:56.883972 19904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:56.883963 19906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:56.883967 19903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:56.884253 19898 server_base.cc:1034] running on GCE node
I20260927 03:31:56.884442 19898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:56.884687 19898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:56.885840 19898 hybrid_clock.cc:648] HybridClock initialized: now 1790479916885830 us; error 28 us; skew 500 ppm
Sep 27 03:31:56 dist-test-slave-j0hf krb5kdc[19523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:31:56.888273 19898 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 03:31:56.888737 19898 webserver.cc:460] Webserver started at http://127.0.98.3:42213/ using document root <none> and password file <none>
I20260927 03:31:56.888991 19898 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:56.889055 19898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:56.889204 19898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:31:56.890285 19898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-2/data/instance:
uuid: "daa8b993264d480c86a80bdea02a861d"
format_stamp: "Formatted at 2026-09-27 03:31:56 on dist-test-slave-j0hf"
server_key: "a97108fac3e0736f9941546089e6bbb3"
server_key_iv: "3b1fb7acf5677bc5b29f828b77af5c93"
server_key_version: "encryptionkey@0"
I20260927 03:31:56.890635 19898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-2/wal/instance:
uuid: "daa8b993264d480c86a80bdea02a861d"
format_stamp: "Formatted at 2026-09-27 03:31:56 on dist-test-slave-j0hf"
server_key: "a97108fac3e0736f9941546089e6bbb3"
server_key_iv: "3b1fb7acf5677bc5b29f828b77af5c93"
server_key_version: "encryptionkey@0"
I20260927 03:31:56.891978 19898 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:31:56.892665 19913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:56.892818 19898 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:31:56.892915 19898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "daa8b993264d480c86a80bdea02a861d"
format_stamp: "Formatted at 2026-09-27 03:31:56 on dist-test-slave-j0hf"
server_key: "a97108fac3e0736f9941546089e6bbb3"
server_key_iv: "3b1fb7acf5677bc5b29f828b77af5c93"
server_key_version: "encryptionkey@0"
I20260927 03:31:56.892992 19898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:56.904313 19898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:56.906252 19898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:56.906410 19898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:56.906632 19898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:31:56.906922 19898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:31:56.906971 19898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:56.907017 19898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:31:56.907043 19898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:56.913215 19898 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43209
I20260927 03:31:56.913256 20026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43209 every 8 connection(s)
I20260927 03:31:56.913575 19898 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:31:56 dist-test-slave-j0hf krb5kdc[19523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:31:56.921077 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 19898
I20260927 03:31:56.921201 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:31:56.921402 392 external_mini_cluster.cc:1428] Setting key 835b22d0e9ca5945b36b7e4aa3cc9199
I20260927 03:31:56.923013 19616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:56.914351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43509 (local address 127.0.98.62:46509)
0927 03:31:56.914401 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:56.914404 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:56.914703 (+ 299us) server_negotiation.cc:407] Connection header received
0927 03:31:56.915792 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:56.915794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:56.915803 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:56.915853 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:56.916459 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:56.916942 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:56.917934 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:56.918137 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:56.918187 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:56.920327 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:56.920329 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:56.920329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:31:56.920336 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:56.921699 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:56.922091 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:56.922092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:56.922093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:56.922152 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:56.922389 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:56.922390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:56.922390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:56.922507 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:56.922572 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:56.922841 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 03:31:56.922921 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:31:56.923216 20029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:56.914306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46509 (local address 127.0.98.3:43509)
0927 03:31:56.914584 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:56.914621 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:56.915696 (+ 1075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:56.915917 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:56.915921 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:56.916180 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:31:56.916382 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:56.916387 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:56.917015 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:56.917016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:56.917859 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:56.917861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:56.918203 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:56.918204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:56.918309 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:56.918783 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:31:56.918791 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:31:56.920227 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:31:56.921787 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:31:56.921794 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:31:56.921796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:31:56.922016 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:31:56.922216 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:31:56.922217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:31:56.922217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:31:56.922319 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:31:56.922578 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:31:56.922584 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:31:56.922715 (+ 131us) client_negotiation.cc:769] Sending connection context
0927 03:31:56.922820 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 03:31:56.922976 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":87,"threads_started":1}
I20260927 03:31:56.923694 20027 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46509
I20260927 03:31:56.923776 20027 heartbeater.cc:464] Registering TS with master...
I20260927 03:31:56.923914 20027 heartbeater.cc:511] Master 127.0.98.62:46509 requested a full tablet report, sending...
I20260927 03:31:56.924245 19568 ts_manager.cc:194] Registered new tserver with Master: daa8b993264d480c86a80bdea02a861d (127.0.98.3:43209)
I20260927 03:31:56.924652 19568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43509
I20260927 03:31:56.933122 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:31:56.940004 19616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:56.934428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37118 (local address 127.0.98.62:46509)
0927 03:31:56.934472 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:56.934479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:56.934598 (+ 119us) server_negotiation.cc:407] Connection header received
0927 03:31:56.934644 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:56.934645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:56.934654 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:56.934706 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:56.935263 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:56.935800 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:56.936494 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:56.936669 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:56.936731 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:56.937841 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:56.937843 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:56.937843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:31:56.937849 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:56.939012 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:56.939311 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:56.939311 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:56.939312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:56.939360 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:56.939580 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:56.939580 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:56.939581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:56.939722 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:56.939798 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:56.939883 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 03:31:56.939940 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:31:56.941617 19568 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:31:56.943295 19568 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:31:56.949935 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:56.945635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57284 (local address 127.0.98.1:40709)
0927 03:31:56.945817 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:56.945823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:56.946201 (+ 378us) server_negotiation.cc:407] Connection header received
0927 03:31:56.946344 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:56.946347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:56.946362 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:56.946413 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:56.946662 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:56.947354 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:56.948911 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:56.949584 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:56.949649 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:56.949696 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:56.949816 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 03:31:56.949854 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":84,"threads_started":1}
I20260927 03:31:56.949965 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:56.945864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43209 (local address 127.0.0.1:52956)
0927 03:31:56.946274 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:56.946285 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:56.946345 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:56.946556 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:56.946557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:56.946558 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:56.946591 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:56.946593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:56.947820 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:56.947821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:56.948712 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:56.948715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:56.949606 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:56.949607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:56.949702 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:56.949719 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 03:31:56.949795 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 03:31:56.949864 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":24,"thread_start_us":46,"threads_started":1}
I20260927 03:31:56.950245 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:56.945946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52956 (local address 127.0.98.3:43209)
0927 03:31:56.946113 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:56.946118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:56.946405 (+ 287us) server_negotiation.cc:407] Connection header received
0927 03:31:56.946448 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:56.946450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:56.946465 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:56.946550 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:56.946654 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:56.947712 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:56.948795 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:56.949517 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:56.949591 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:56.949659 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:56.950107 (+ 448us) server_negotiation.cc:299] Negotiation successful
0927 03:31:56.950167 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":106,"threads_started":1}
I20260927 03:31:56.949965 20038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:56.945534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40709 (local address 127.0.0.1:57284)
0927 03:31:56.946132 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:56.946142 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:56.946233 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:56.946523 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:56.946523 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:56.946525 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:56.946580 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:56.946581 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:56.947461 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:56.947462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:56.948833 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:56.948836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:56.949661 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:56.949662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:56.949712 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:56.949719 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 03:31:56.949795 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 03:31:56.949865 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":8,"thread_start_us":223,"threads_started":1}
I20260927 03:31:56.950803 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:56.945528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45135 (local address 127.0.0.1:53594)
0927 03:31:56.945689 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:56.945744 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:56.945816 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:56.946174 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:56.946177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:56.946180 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:56.946255 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:56.946257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:56.947661 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:56.947662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:56.948834 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:56.948836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:56.950579 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:56.950580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:56.950646 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:56.950650 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:31:56.950682 (+ 32us) client_negotiation.cc:240] Negotiation successful
0927 03:31:56.950724 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":11,"thread_start_us":117,"threads_started":1}
I20260927 03:31:56.950990 19961 tablet_service.cc:1467] Processing CreateTablet for tablet 5e4679afa6414c0dbaace74c04c025cd (DEFAULT_TABLE table=test-table [id=4bd565260c1c4783a828eeb70775621b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:31:56.951170 20039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:56.945839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53594 (local address 127.0.98.2:45135)
0927 03:31:56.946017 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:56.946021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:56.946030 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:31:56.946062 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:56.946063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:56.946076 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:56.946169 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:56.946359 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:56.947557 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:56.949737 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:56.950485 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:56.950950 (+ 465us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:56.951028 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:56.951054 (+ 26us) server_negotiation.cc:299] Negotiation successful
0927 03:31:56.951095 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":69,"threads_started":1}
I20260927 03:31:56.951170 19693 tablet_service.cc:1467] Processing CreateTablet for tablet 5e4679afa6414c0dbaace74c04c025cd (DEFAULT_TABLE table=test-table [id=4bd565260c1c4783a828eeb70775621b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:31:56.951357 19961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e4679afa6414c0dbaace74c04c025cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:31:56.951555 19693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e4679afa6414c0dbaace74c04c025cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:31:56.952023 19827 tablet_service.cc:1467] Processing CreateTablet for tablet 5e4679afa6414c0dbaace74c04c025cd (DEFAULT_TABLE table=test-table [id=4bd565260c1c4783a828eeb70775621b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:31:56.952351 19827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e4679afa6414c0dbaace74c04c025cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:31:56.954244 20041 tablet_bootstrap.cc:492] T 5e4679afa6414c0dbaace74c04c025cd P ffe1b22521304b9eb8f053784d407b9d: Bootstrap starting.
I20260927 03:31:56.954752 20042 tablet_bootstrap.cc:492] T 5e4679afa6414c0dbaace74c04c025cd P daa8b993264d480c86a80bdea02a861d: Bootstrap starting.
I20260927 03:31:56.954882 20041 tablet_bootstrap.cc:654] T 5e4679afa6414c0dbaace74c04c025cd P ffe1b22521304b9eb8f053784d407b9d: Neither blocks nor log segments found. Creating new log.
I20260927 03:31:56.955452 20043 tablet_bootstrap.cc:492] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a: Bootstrap starting.
I20260927 03:31:56.955504 20041 log.cc:824] T 5e4679afa6414c0dbaace74c04c025cd P ffe1b22521304b9eb8f053784d407b9d: Log is configured to *not* fsync() on all Append() calls
I20260927 03:31:56.955560 20042 tablet_bootstrap.cc:654] T 5e4679afa6414c0dbaace74c04c025cd P daa8b993264d480c86a80bdea02a861d: Neither blocks nor log segments found. Creating new log.
I20260927 03:31:56.956046 20043 tablet_bootstrap.cc:654] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a: Neither blocks nor log segments found. Creating new log.
I20260927 03:31:56.956049 20042 log.cc:824] T 5e4679afa6414c0dbaace74c04c025cd P daa8b993264d480c86a80bdea02a861d: Log is configured to *not* fsync() on all Append() calls
I20260927 03:31:56.956664 20043 log.cc:824] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a: Log is configured to *not* fsync() on all Append() calls
I20260927 03:31:56.956875 20042 tablet_bootstrap.cc:492] T 5e4679afa6414c0dbaace74c04c025cd P daa8b993264d480c86a80bdea02a861d: No bootstrap required, opened a new log
I20260927 03:31:56.956952 20042 ts_tablet_manager.cc:1397] T 5e4679afa6414c0dbaace74c04c025cd P daa8b993264d480c86a80bdea02a861d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:31:56.957422 20041 tablet_bootstrap.cc:492] T 5e4679afa6414c0dbaace74c04c025cd P ffe1b22521304b9eb8f053784d407b9d: No bootstrap required, opened a new log
I20260927 03:31:56.957453 20043 tablet_bootstrap.cc:492] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a: No bootstrap required, opened a new log
I20260927 03:31:56.957494 20041 ts_tablet_manager.cc:1397] T 5e4679afa6414c0dbaace74c04c025cd P ffe1b22521304b9eb8f053784d407b9d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 03:31:56.957510 20043 ts_tablet_manager.cc:1397] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:31:56.958429 20042 raft_consensus.cc:348] T 5e4679afa6414c0dbaace74c04c025cd P daa8b993264d480c86a80bdea02a861d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe1b22521304b9eb8f053784d407b9d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40709 } } peers { permanent_uuid: "02f1ff87e47c4d59918edb01ec8dd82a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45135 } } peers { permanent_uuid: "daa8b993264d480c86a80bdea02a861d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43209 } }
I20260927 03:31:56.958598 20042 raft_consensus.cc:374] T 5e4679afa6414c0dbaace74c04c025cd P daa8b993264d480c86a80bdea02a861d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:31:56.958642 20042 raft_consensus.cc:729] T 5e4679afa6414c0dbaace74c04c025cd P daa8b993264d480c86a80bdea02a861d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daa8b993264d480c86a80bdea02a861d, State: Initialized, Role: FOLLOWER
I20260927 03:31:56.958771 20042 consensus_queue.cc:260] T 5e4679afa6414c0dbaace74c04c025cd P daa8b993264d480c86a80bdea02a861d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe1b22521304b9eb8f053784d407b9d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40709 } } peers { permanent_uuid: "02f1ff87e47c4d59918edb01ec8dd82a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45135 } } peers { permanent_uuid: "daa8b993264d480c86a80bdea02a861d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43209 } }
I20260927 03:31:56.959028 20042 ts_tablet_manager.cc:1428] T 5e4679afa6414c0dbaace74c04c025cd P daa8b993264d480c86a80bdea02a861d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 03:31:56.959161 20027 heartbeater.cc:503] Master 127.0.98.62:46509 was elected leader, sending a full tablet report...
I20260927 03:31:56.959481 20041 raft_consensus.cc:348] T 5e4679afa6414c0dbaace74c04c025cd P ffe1b22521304b9eb8f053784d407b9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe1b22521304b9eb8f053784d407b9d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40709 } } peers { permanent_uuid: "02f1ff87e47c4d59918edb01ec8dd82a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45135 } } peers { permanent_uuid: "daa8b993264d480c86a80bdea02a861d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43209 } }
I20260927 03:31:56.959487 20043 raft_consensus.cc:348] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe1b22521304b9eb8f053784d407b9d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40709 } } peers { permanent_uuid: "02f1ff87e47c4d59918edb01ec8dd82a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45135 } } peers { permanent_uuid: "daa8b993264d480c86a80bdea02a861d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43209 } }
I20260927 03:31:56.959653 20043 raft_consensus.cc:374] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:31:56.959653 20041 raft_consensus.cc:374] T 5e4679afa6414c0dbaace74c04c025cd P ffe1b22521304b9eb8f053784d407b9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:31:56.959692 20041 raft_consensus.cc:729] T 5e4679afa6414c0dbaace74c04c025cd P ffe1b22521304b9eb8f053784d407b9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffe1b22521304b9eb8f053784d407b9d, State: Initialized, Role: FOLLOWER
I20260927 03:31:56.959692 20043 raft_consensus.cc:729] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02f1ff87e47c4d59918edb01ec8dd82a, State: Initialized, Role: FOLLOWER
I20260927 03:31:56.959823 20041 consensus_queue.cc:260] T 5e4679afa6414c0dbaace74c04c025cd P ffe1b22521304b9eb8f053784d407b9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe1b22521304b9eb8f053784d407b9d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40709 } } peers { permanent_uuid: "02f1ff87e47c4d59918edb01ec8dd82a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45135 } } peers { permanent_uuid: "daa8b993264d480c86a80bdea02a861d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43209 } }
I20260927 03:31:56.959823 20043 consensus_queue.cc:260] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe1b22521304b9eb8f053784d407b9d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40709 } } peers { permanent_uuid: "02f1ff87e47c4d59918edb01ec8dd82a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45135 } } peers { permanent_uuid: "daa8b993264d480c86a80bdea02a861d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43209 } }
I20260927 03:31:56.960103 20041 ts_tablet_manager.cc:1428] T 5e4679afa6414c0dbaace74c04c025cd P ffe1b22521304b9eb8f053784d407b9d: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260927 03:31:56.960103 20043 ts_tablet_manager.cc:1428] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:31:56.960228 19759 heartbeater.cc:503] Master 127.0.98.62:46509 was elected leader, sending a full tablet report...
I20260927 03:31:56.960475 19893 heartbeater.cc:503] Master 127.0.98.62:46509 was elected leader, sending a full tablet report...
W20260927 03:31:57.013674 19894 tablet.cc:2369] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:31:57.070433 20048 raft_consensus.cc:482] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:31:57.070559 20048 raft_consensus.cc:504] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe1b22521304b9eb8f053784d407b9d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40709 } } peers { permanent_uuid: "02f1ff87e47c4d59918edb01ec8dd82a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45135 } } peers { permanent_uuid: "daa8b993264d480c86a80bdea02a861d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43209 } }
I20260927 03:31:57.070904 20048 leader_election.cc:288] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffe1b22521304b9eb8f053784d407b9d (127.0.98.1:40709), daa8b993264d480c86a80bdea02a861d (127.0.98.3:43209)
I20260927 03:31:57.074718 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:57.071200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44635 (local address 127.0.98.1:40709)
0927 03:31:57.071249 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:57.071252 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:57.071263 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:31:57.071293 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:57.071294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:57.071306 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:57.071372 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:57.071561 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.072325 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.073787 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.074345 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.074421 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:57.074431 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:57.074605 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 03:31:57.074643 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 03:31:57.074718 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:57.071265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51145 (local address 127.0.98.3:43209)
0927 03:31:57.071316 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:57.071319 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:57.071473 (+ 154us) server_negotiation.cc:407] Connection header received
0927 03:31:57.071544 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:57.071545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:57.071555 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:57.071610 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:57.071735 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.072407 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.073786 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.074345 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.074421 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:57.074431 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:57.074605 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 03:31:57.074644 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:31:57.074775 19895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:57.071079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40709 (local address 127.0.98.2:44635)
0927 03:31:57.071144 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:57.071159 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:57.071221 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:57.071372 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:57.071373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:57.071375 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:57.071483 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:57.071485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.072412 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.072412 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:57.073696 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:57.073700 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.074430 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.074431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:57.074529 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:57.074533 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:31:57.074591 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 03:31:57.074651 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":87}
I20260927 03:31:57.074987 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:57.071189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43209 (local address 127.0.98.2:51145)
0927 03:31:57.071379 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:57.071389 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:57.071473 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:57.071620 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:57.071621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:57.071622 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:57.071671 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:57.071672 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.072496 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.072497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:57.073696 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:57.073700 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.074431 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.074431 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:57.074529 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:57.074533 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:31:57.074591 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 03:31:57.074651 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":13,"thread_start_us":127,"threads_started":1}
I20260927 03:31:57.075146 19713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4679afa6414c0dbaace74c04c025cd" candidate_uuid: "02f1ff87e47c4d59918edb01ec8dd82a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe1b22521304b9eb8f053784d407b9d" is_pre_election: true
I20260927 03:31:57.075350 19713 raft_consensus.cc:2454] T 5e4679afa6414c0dbaace74c04c025cd P ffe1b22521304b9eb8f053784d407b9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02f1ff87e47c4d59918edb01ec8dd82a in term 0.
I20260927 03:31:57.075415 19981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4679afa6414c0dbaace74c04c025cd" candidate_uuid: "02f1ff87e47c4d59918edb01ec8dd82a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daa8b993264d480c86a80bdea02a861d" is_pre_election: true
I20260927 03:31:57.075610 19981 raft_consensus.cc:2454] T 5e4679afa6414c0dbaace74c04c025cd P daa8b993264d480c86a80bdea02a861d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02f1ff87e47c4d59918edb01ec8dd82a in term 0.
I20260927 03:31:57.075824 19781 leader_election.cc:302] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02f1ff87e47c4d59918edb01ec8dd82a, daa8b993264d480c86a80bdea02a861d; no voters:
I20260927 03:31:57.075978 20048 raft_consensus.cc:2781] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:31:57.076071 20048 raft_consensus.cc:482] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:31:57.076193 20048 raft_consensus.cc:3037] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:31:57.076930 20048 raft_consensus.cc:504] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe1b22521304b9eb8f053784d407b9d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40709 } } peers { permanent_uuid: "02f1ff87e47c4d59918edb01ec8dd82a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45135 } } peers { permanent_uuid: "daa8b993264d480c86a80bdea02a861d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43209 } }
I20260927 03:31:57.077054 20048 leader_election.cc:288] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a [CANDIDATE]: Term 1 election: Requested vote from peers ffe1b22521304b9eb8f053784d407b9d (127.0.98.1:40709), daa8b993264d480c86a80bdea02a861d (127.0.98.3:43209)
I20260927 03:31:57.077193 19713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4679afa6414c0dbaace74c04c025cd" candidate_uuid: "02f1ff87e47c4d59918edb01ec8dd82a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe1b22521304b9eb8f053784d407b9d"
I20260927 03:31:57.077232 19981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4679afa6414c0dbaace74c04c025cd" candidate_uuid: "02f1ff87e47c4d59918edb01ec8dd82a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daa8b993264d480c86a80bdea02a861d"
I20260927 03:31:57.077291 19713 raft_consensus.cc:3037] T 5e4679afa6414c0dbaace74c04c025cd P ffe1b22521304b9eb8f053784d407b9d [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:31:57.077306 19981 raft_consensus.cc:3037] T 5e4679afa6414c0dbaace74c04c025cd P daa8b993264d480c86a80bdea02a861d [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:31:57.077994 19713 raft_consensus.cc:2454] T 5e4679afa6414c0dbaace74c04c025cd P ffe1b22521304b9eb8f053784d407b9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02f1ff87e47c4d59918edb01ec8dd82a in term 1.
I20260927 03:31:57.078016 19981 raft_consensus.cc:2454] T 5e4679afa6414c0dbaace74c04c025cd P daa8b993264d480c86a80bdea02a861d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02f1ff87e47c4d59918edb01ec8dd82a in term 1.
I20260927 03:31:57.078181 19781 leader_election.cc:302] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02f1ff87e47c4d59918edb01ec8dd82a, daa8b993264d480c86a80bdea02a861d; no voters:
I20260927 03:31:57.078279 20048 raft_consensus.cc:2781] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:31:57.078423 20048 raft_consensus.cc:686] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a [term 1 LEADER]: Becoming Leader. State: Replica: 02f1ff87e47c4d59918edb01ec8dd82a, State: Running, Role: LEADER
I20260927 03:31:57.078511 20048 consensus_queue.cc:237] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe1b22521304b9eb8f053784d407b9d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40709 } } peers { permanent_uuid: "02f1ff87e47c4d59918edb01ec8dd82a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45135 } } peers { permanent_uuid: "daa8b993264d480c86a80bdea02a861d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43209 } }
I20260927 03:31:57.079385 19567 catalog_manager.cc:5495] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a reported cstate change: term changed from 0 to 1, leader changed from <none> to 02f1ff87e47c4d59918edb01ec8dd82a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "02f1ff87e47c4d59918edb01ec8dd82a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe1b22521304b9eb8f053784d407b9d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02f1ff87e47c4d59918edb01ec8dd82a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "daa8b993264d480c86a80bdea02a861d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43209 } health_report { overall_health: UNKNOWN } } }
I20260927 03:31:57.093544 20039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:57.091066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53608 (local address 127.0.98.2:45135)
0927 03:31:57.091125 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:57.091127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:57.091136 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:31:57.091223 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:57.091224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:57.091244 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:57.091320 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:31:57.091482 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.091978 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.092743 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.092962 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.093055 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:57.093213 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:31:57.093314 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:31:57.093403 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:57.093422 (+ 19us) server_negotiation.cc:299] Negotiation successful
0927 03:31:57.093442 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 03:31:57.095088 19760 tablet.cc:2369] T 5e4679afa6414c0dbaace74c04c025cd P ffe1b22521304b9eb8f053784d407b9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:31:57.095325 19981 raft_consensus.cc:1261] T 5e4679afa6414c0dbaace74c04c025cd P daa8b993264d480c86a80bdea02a861d [term 1 FOLLOWER]: Refusing update from remote peer 02f1ff87e47c4d59918edb01ec8dd82a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:31:57.095325 19713 raft_consensus.cc:1261] T 5e4679afa6414c0dbaace74c04c025cd P ffe1b22521304b9eb8f053784d407b9d [term 1 FOLLOWER]: Refusing update from remote peer 02f1ff87e47c4d59918edb01ec8dd82a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:31:57.095682 20048 consensus_queue.cc:1035] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffe1b22521304b9eb8f053784d407b9d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:31:57.095808 20051 consensus_queue.cc:1035] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a [LEADER]: Connected to new peer: Peer: permanent_uuid: "daa8b993264d480c86a80bdea02a861d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:31:57.098182 20055 mvcc.cc:205] Tried to move back new op lower bound from 7333805740419452928 to 7333805740354232320. Current Snapshot: MvccSnapshot[applied={T|T < 7333805740419452928}]
I20260927 03:31:57.098199 20056 mvcc.cc:205] Tried to move back new op lower bound from 7333805740419452928 to 7333805740354232320. Current Snapshot: MvccSnapshot[applied={T|T < 7333805740419452928}]
I20260927 03:31:57.098652 20054 mvcc.cc:205] Tried to move back new op lower bound from 7333805740419452928 to 7333805740354232320. Current Snapshot: MvccSnapshot[applied={T|T < 7333805740419452928 or (T in {7333805740419452928})}]
I20260927 03:31:57.100268 19567 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:31:57.100360 19567 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:31:57.101102 19567 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e576a32023e7459e8d84b6834eb559c0: Sending DeleteTablet for 3 replicas of tablet 5e4679afa6414c0dbaace74c04c025cd
I20260927 03:31:57.101503 19693 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e4679afa6414c0dbaace74c04c025cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:31:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57284
I20260927 03:31:57.101541 19961 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e4679afa6414c0dbaace74c04c025cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:31:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52956
I20260927 03:31:57.101532 19827 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e4679afa6414c0dbaace74c04c025cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:31:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53594
I20260927 03:31:57.101722 20066 tablet_replica.cc:330] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a: stopping tablet replica
I20260927 03:31:57.101810 20066 raft_consensus.cc:2229] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:31:57.101903 20066 raft_consensus.cc:2258] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:31:57.101984 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 19630
I20260927 03:31:57.102342 20066 ts_tablet_manager.cc:1905] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:31:57.104198 20066 ts_tablet_manager.cc:1918] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:31:57.104285 20066 log.cc:1196] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-1/wal/wals/5e4679afa6414c0dbaace74c04c025cd
I20260927 03:31:57.104568 20066 ts_tablet_manager.cc:1939] T 5e4679afa6414c0dbaace74c04c025cd P 02f1ff87e47c4d59918edb01ec8dd82a: Deleting consensus metadata
I20260927 03:31:57.104959 19555 catalog_manager.cc:4841] TS 02f1ff87e47c4d59918edb01ec8dd82a (127.0.98.2:45135): tablet 5e4679afa6414c0dbaace74c04c025cd (table test-table [id=4bd565260c1c4783a828eeb70775621b]) successfully deleted
W20260927 03:31:57.110386 19553 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40709 (error 108)
I20260927 03:31:57.110432 20065 tablet_replica.cc:330] T 5e4679afa6414c0dbaace74c04c025cd P daa8b993264d480c86a80bdea02a861d: stopping tablet replica
I20260927 03:31:57.110529 20065 raft_consensus.cc:2229] T 5e4679afa6414c0dbaace74c04c025cd P daa8b993264d480c86a80bdea02a861d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:31:57.110648 20065 raft_consensus.cc:2258] T 5e4679afa6414c0dbaace74c04c025cd P daa8b993264d480c86a80bdea02a861d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:31:57.110899 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:57.110781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40709 (local address 127.0.0.1:57300)
0927 03:31:57.110826 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:57.110864 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 03:31:57.111020 19553 catalog_manager.cc:4584] TS ffe1b22521304b9eb8f053784d407b9d (127.0.98.1:40709): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5e4679afa6414c0dbaace74c04c025cd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40709: connect: Connection refused (error 111)
I20260927 03:31:57.111114 20065 ts_tablet_manager.cc:1905] T 5e4679afa6414c0dbaace74c04c025cd P daa8b993264d480c86a80bdea02a861d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:31:57.111373 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 19764
I20260927 03:31:57.112964 20065 ts_tablet_manager.cc:1918] T 5e4679afa6414c0dbaace74c04c025cd P daa8b993264d480c86a80bdea02a861d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:31:57.113044 20065 log.cc:1196] T 5e4679afa6414c0dbaace74c04c025cd P daa8b993264d480c86a80bdea02a861d: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790479675865103-392-0/minicluster-data/ts-2/wal/wals/5e4679afa6414c0dbaace74c04c025cd
I20260927 03:31:57.113364 20065 ts_tablet_manager.cc:1939] T 5e4679afa6414c0dbaace74c04c025cd P daa8b993264d480c86a80bdea02a861d: Deleting consensus metadata
I20260927 03:31:57.113838 19553 catalog_manager.cc:4841] TS daa8b993264d480c86a80bdea02a861d (127.0.98.3:43209): tablet 5e4679afa6414c0dbaace74c04c025cd (table test-table [id=4bd565260c1c4783a828eeb70775621b]) successfully deleted
I20260927 03:31:57.116605 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 19898
I20260927 03:31:57.120995 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 19536
2026-09-27T03:31:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1999 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T03:31:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:31:57Z Disabled control of system clock
I20260927 03:31:57.148306 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35011
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 03:31:57.227850 20071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:57.228096 20071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:57.228116 20071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:57.229328 20071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:31:57.229374 20071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:57.229387 20071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:57.229400 20071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:31:57.229413 20071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:31:57.230789 20071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35011
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:57.231060 20071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:57.231367 20071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:31:57.236219 20076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:57.236213 20079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:57.236196 20077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:57.236461 20071 server_base.cc:1034] running on GCE node
I20260927 03:31:57.236730 20071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:57.236969 20071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:57.238131 20071 hybrid_clock.cc:648] HybridClock initialized: now 1790479917238119 us; error 29 us; skew 500 ppm
I20260927 03:31:57.239303 20071 webserver.cc:460] Webserver started at http://127.0.98.62:34801/ using document root <none> and password file <none>
I20260927 03:31:57.239609 20071 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:57.239675 20071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:57.239809 20071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:31:57.240868 20071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "f57cbec513e64c6698cbdcc071015779"
format_stamp: "Formatted at 2026-09-27 03:31:57 on dist-test-slave-j0hf"
server_key: "9e5b443287b7e79fa066f39a45e73474"
server_key_iv: "0cccb5068c74ab970ee8b420b301d83b"
server_key_version: "encryptionkey@0"
I20260927 03:31:57.241262 20071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "f57cbec513e64c6698cbdcc071015779"
format_stamp: "Formatted at 2026-09-27 03:31:57 on dist-test-slave-j0hf"
server_key: "9e5b443287b7e79fa066f39a45e73474"
server_key_iv: "0cccb5068c74ab970ee8b420b301d83b"
server_key_version: "encryptionkey@0"
I20260927 03:31:57.242565 20071 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:31:57.243297 20085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:57.243515 20071 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:31:57.243602 20071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "f57cbec513e64c6698cbdcc071015779"
format_stamp: "Formatted at 2026-09-27 03:31:57 on dist-test-slave-j0hf"
server_key: "9e5b443287b7e79fa066f39a45e73474"
server_key_iv: "0cccb5068c74ab970ee8b420b301d83b"
server_key_version: "encryptionkey@0"
I20260927 03:31:57.243680 20071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:57.254293 20071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:57.254582 20071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:57.254724 20071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:57.258538 20071 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35011
I20260927 03:31:57.258574 20137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35011 every 8 connection(s)
I20260927 03:31:57.258919 20071 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:31:57.259935 20138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:31:57.262177 20138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f57cbec513e64c6698cbdcc071015779: Bootstrap starting.
I20260927 03:31:57.262297 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 20071
I20260927 03:31:57.262408 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:31:57.262594 392 external_mini_cluster.cc:1428] Setting key b4716e18ad9dcdb58a4cd9b06fcd1e5e
I20260927 03:31:57.262840 20138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f57cbec513e64c6698cbdcc071015779: Neither blocks nor log segments found. Creating new log.
I20260927 03:31:57.263180 20138 log.cc:824] T 00000000000000000000000000000000 P f57cbec513e64c6698cbdcc071015779: Log is configured to *not* fsync() on all Append() calls
I20260927 03:31:57.263880 20138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f57cbec513e64c6698cbdcc071015779: No bootstrap required, opened a new log
I20260927 03:31:57.265657 20138 raft_consensus.cc:348] T 00000000000000000000000000000000 P f57cbec513e64c6698cbdcc071015779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f57cbec513e64c6698cbdcc071015779" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35011 } }
I20260927 03:31:57.265816 20138 raft_consensus.cc:374] T 00000000000000000000000000000000 P f57cbec513e64c6698cbdcc071015779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:31:57.265863 20138 raft_consensus.cc:729] T 00000000000000000000000000000000 P f57cbec513e64c6698cbdcc071015779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f57cbec513e64c6698cbdcc071015779, State: Initialized, Role: FOLLOWER
I20260927 03:31:57.266019 20138 consensus_queue.cc:260] T 00000000000000000000000000000000 P f57cbec513e64c6698cbdcc071015779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f57cbec513e64c6698cbdcc071015779" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35011 } }
I20260927 03:31:57.266099 20138 raft_consensus.cc:388] T 00000000000000000000000000000000 P f57cbec513e64c6698cbdcc071015779 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:31:57.266127 20138 raft_consensus.cc:482] T 00000000000000000000000000000000 P f57cbec513e64c6698cbdcc071015779 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:31:57.266170 20138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f57cbec513e64c6698cbdcc071015779 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:31:57.266933 20138 raft_consensus.cc:504] T 00000000000000000000000000000000 P f57cbec513e64c6698cbdcc071015779 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f57cbec513e64c6698cbdcc071015779" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35011 } }
I20260927 03:31:57.267041 20138 leader_election.cc:302] T 00000000000000000000000000000000 P f57cbec513e64c6698cbdcc071015779 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f57cbec513e64c6698cbdcc071015779; no voters:
I20260927 03:31:57.267212 20138 leader_election.cc:288] T 00000000000000000000000000000000 P f57cbec513e64c6698cbdcc071015779 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:31:57.267253 20143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f57cbec513e64c6698cbdcc071015779 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:31:57.267481 20143 raft_consensus.cc:686] T 00000000000000000000000000000000 P f57cbec513e64c6698cbdcc071015779 [term 1 LEADER]: Becoming Leader. State: Replica: f57cbec513e64c6698cbdcc071015779, State: Running, Role: LEADER
I20260927 03:31:57.267560 20138 sys_catalog.cc:564] T 00000000000000000000000000000000 P f57cbec513e64c6698cbdcc071015779 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:31:57.267613 20143 consensus_queue.cc:237] T 00000000000000000000000000000000 P f57cbec513e64c6698cbdcc071015779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f57cbec513e64c6698cbdcc071015779" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35011 } }
I20260927 03:31:57.268203 20141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:57.263092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60822 (local address 127.0.98.62:35011)
0927 03:31:57.263288 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:57.263297 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:57.263329 (+ 32us) server_negotiation.cc:407] Connection header received
0927 03:31:57.263789 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:57.263797 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:57.263855 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:57.264009 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:57.264233 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.265124 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.265923 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.266539 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.266695 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:57.267382 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:57.267387 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:57.267387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:31:57.267633 (+ 246us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:57.267676 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:31:57.267691 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:57.267757 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:57.267876 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 03:31:57.267965 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260927 03:31:57.268075 20144 sys_catalog.cc:455] T 00000000000000000000000000000000 P f57cbec513e64c6698cbdcc071015779 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f57cbec513e64c6698cbdcc071015779" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f57cbec513e64c6698cbdcc071015779" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35011 } } }
I20260927 03:31:57.268560 20144 sys_catalog.cc:458] T 00000000000000000000000000000000 P f57cbec513e64c6698cbdcc071015779 [sys.catalog]: This master's current role is: LEADER
I20260927 03:31:57.268061 20143 sys_catalog.cc:455] T 00000000000000000000000000000000 P f57cbec513e64c6698cbdcc071015779 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f57cbec513e64c6698cbdcc071015779. Latest consensus state: current_term: 1 leader_uuid: "f57cbec513e64c6698cbdcc071015779" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f57cbec513e64c6698cbdcc071015779" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35011 } } }
I20260927 03:31:57.268908 20143 sys_catalog.cc:458] T 00000000000000000000000000000000 P f57cbec513e64c6698cbdcc071015779 [sys.catalog]: This master's current role is: LEADER
I20260927 03:31:57.269006 20154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:31:57.269567 20154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:31:57.270934 20154 catalog_manager.cc:1347] Generated new cluster ID: bb5cebd068c8447a9db471d64d663abc
I20260927 03:31:57.271001 20154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:31:57.295742 20154 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:31:57.296501 20154 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:31:57.300905 20154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f57cbec513e64c6698cbdcc071015779: Generated new TSK 0
I20260927 03:31:57.301131 20154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:31:57.309712 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35011
--builtin_ntp_servers=127.0.98.20:41333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 03:31:57.391173 20162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:57.391377 20162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:57.391399 20162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:57.392583 20162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:57.392627 20162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:57.392652 20162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 03:31:57.394034 20162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:57.394273 20162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:57.394575 20162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:31:57.399601 20170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:57.399596 20167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:57.399636 20162 server_base.cc:1034] running on GCE node
W20260927 03:31:57.399601 20168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:57.400074 20162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:57.400331 20162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:57.401471 20162 hybrid_clock.cc:648] HybridClock initialized: now 1790479917401453 us; error 37 us; skew 500 ppm
I20260927 03:31:57.402691 20162 webserver.cc:460] Webserver started at http://127.0.98.1:41787/ using document root <none> and password file <none>
I20260927 03:31:57.402936 20162 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:57.403007 20162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:57.403163 20162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:31:57.404361 20162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-0/data/instance:
uuid: "460444f334f34b11af91ddef6590f344"
format_stamp: "Formatted at 2026-09-27 03:31:57 on dist-test-slave-j0hf"
server_key: "2098692aea25c0a651a098b96d7f8b03"
server_key_iv: "fa80ee7ac15c163440b199e30832f911"
server_key_version: "encryptionkey@0"
I20260927 03:31:57.404744 20162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-0/wal/instance:
uuid: "460444f334f34b11af91ddef6590f344"
format_stamp: "Formatted at 2026-09-27 03:31:57 on dist-test-slave-j0hf"
server_key: "2098692aea25c0a651a098b96d7f8b03"
server_key_iv: "fa80ee7ac15c163440b199e30832f911"
server_key_version: "encryptionkey@0"
I20260927 03:31:57.406082 20162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:31:57.406865 20176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:57.407049 20162 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:31:57.407146 20162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-0/wal
uuid: "460444f334f34b11af91ddef6590f344"
format_stamp: "Formatted at 2026-09-27 03:31:57 on dist-test-slave-j0hf"
server_key: "2098692aea25c0a651a098b96d7f8b03"
server_key_iv: "fa80ee7ac15c163440b199e30832f911"
server_key_version: "encryptionkey@0"
I20260927 03:31:57.407224 20162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:57.427358 20162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:57.427729 20162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:57.427889 20162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:57.428143 20162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:31:57.428481 20162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:31:57.428541 20162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:57.428582 20162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:31:57.428619 20162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:57.434528 20162 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33043
I20260927 03:31:57.434576 20289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33043 every 8 connection(s)
I20260927 03:31:57.434924 20162 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-0/data/info.pb
I20260927 03:31:57.440029 20141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:57.435706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60307 (local address 127.0.98.62:35011)
0927 03:31:57.435789 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:57.435791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:57.436009 (+ 218us) server_negotiation.cc:407] Connection header received
0927 03:31:57.436495 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:57.436496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:57.436506 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:57.436557 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:57.437224 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.437715 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.438687 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.438870 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.438920 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:57.439830 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:57.439831 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:57.439831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:31:57.439836 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:57.439848 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:31:57.439852 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:57.439887 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:57.439944 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 03:31:57.439965 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 03:31:57.440248 20292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:57.435686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35011 (local address 127.0.98.1:60307)
0927 03:31:57.435889 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:57.435924 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:57.436413 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:57.436621 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:57.436624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:57.436632 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:31:57.437152 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:57.437157 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.437799 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.437799 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:57.438612 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:57.438614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.438942 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.438943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:57.439060 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:57.439696 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:31:57.439697 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:31:57.439728 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:31:57.439751 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:31:57.439888 (+ 137us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:31:57.439890 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:31:57.439892 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:31:57.439940 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 03:31:57.440026 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":103,"threads_started":1}
I20260927 03:31:57.440871 20290 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35011
I20260927 03:31:57.440954 20290 heartbeater.cc:464] Registering TS with master...
I20260927 03:31:57.441116 20290 heartbeater.cc:511] Master 127.0.98.62:35011 requested a full tablet report, sending...
I20260927 03:31:57.441562 20102 ts_manager.cc:194] Registered new tserver with Master: 460444f334f34b11af91ddef6590f344 (127.0.98.1:33043)
I20260927 03:31:57.442229 20102 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:60307
I20260927 03:31:57.444689 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 20162
I20260927 03:31:57.444799 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-0/wal/instance
I20260927 03:31:57.445012 392 external_mini_cluster.cc:1428] Setting key 0ab24300c00fea8c7b8ab2934755a129
I20260927 03:31:57.445914 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35011
--builtin_ntp_servers=127.0.98.20:41333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 03:31:57.525461 20293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:57.525667 20293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:57.525691 20293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:57.526849 20293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:57.526894 20293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:57.526919 20293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 03:31:57.528303 20293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:57.528553 20293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:57.528841 20293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:31:57.533664 20298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:57.533681 20299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:57.533696 20293 server_base.cc:1034] running on GCE node
W20260927 03:31:57.533674 20301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:57.534039 20293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:57.534276 20293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:57.535452 20293 hybrid_clock.cc:648] HybridClock initialized: now 1790479917535436 us; error 45 us; skew 500 ppm
I20260927 03:31:57.536594 20293 webserver.cc:460] Webserver started at http://127.0.98.2:38931/ using document root <none> and password file <none>
I20260927 03:31:57.536849 20293 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:57.536912 20293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:57.537063 20293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:31:57.538064 20293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-1/data/instance:
uuid: "2f8ded2ae59c40ebaa78644bcdc5c757"
format_stamp: "Formatted at 2026-09-27 03:31:57 on dist-test-slave-j0hf"
server_key: "81b44fbdbf9a585ca7d3cfbd2947da34"
server_key_iv: "738064dd9f4dc04441dad40638c59a69"
server_key_version: "encryptionkey@0"
I20260927 03:31:57.538415 20293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2f8ded2ae59c40ebaa78644bcdc5c757"
format_stamp: "Formatted at 2026-09-27 03:31:57 on dist-test-slave-j0hf"
server_key: "81b44fbdbf9a585ca7d3cfbd2947da34"
server_key_iv: "738064dd9f4dc04441dad40638c59a69"
server_key_version: "encryptionkey@0"
I20260927 03:31:57.539692 20293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:31:57.540366 20307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:57.540531 20293 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:31:57.540623 20293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-1/wal
uuid: "2f8ded2ae59c40ebaa78644bcdc5c757"
format_stamp: "Formatted at 2026-09-27 03:31:57 on dist-test-slave-j0hf"
server_key: "81b44fbdbf9a585ca7d3cfbd2947da34"
server_key_iv: "738064dd9f4dc04441dad40638c59a69"
server_key_version: "encryptionkey@0"
I20260927 03:31:57.540702 20293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:57.551084 20293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:57.551362 20293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:57.551527 20293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:57.551735 20293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:31:57.552021 20293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:31:57.552071 20293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:57.552109 20293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:31:57.552134 20293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:57.557941 20293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45951
I20260927 03:31:57.558014 20420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45951 every 8 connection(s)
I20260927 03:31:57.558341 20293 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-1/data/info.pb
I20260927 03:31:57.560592 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 20293
I20260927 03:31:57.560729 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-1/wal/instance
I20260927 03:31:57.560956 392 external_mini_cluster.cc:1428] Setting key ab9e659795b072768df9e597036df01e
I20260927 03:31:57.561995 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35011
--builtin_ntp_servers=127.0.98.20:41333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 03:31:57.564638 20141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:57.559055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46833 (local address 127.0.98.62:35011)
0927 03:31:57.559116 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:57.559119 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:57.559341 (+ 222us) server_negotiation.cc:407] Connection header received
0927 03:31:57.559952 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:57.559953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:57.559967 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:57.560027 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:57.560746 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.561494 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.563072 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.563264 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.563346 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:57.564336 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:57.564338 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:57.564338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:31:57.564343 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:57.564358 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:31:57.564363 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:57.564411 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:57.564525 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 03:31:57.564555 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 03:31:57.564879 20423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:57.559055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35011 (local address 127.0.98.2:46833)
0927 03:31:57.559219 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:57.559257 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:57.559821 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:57.560097 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:57.560102 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:57.560117 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:31:57.560610 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:57.560615 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.561587 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.561588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:57.562987 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:57.562990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.563353 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.563354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:57.563569 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:57.564144 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:31:57.564144 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:31:57.564174 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:31:57.564197 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:31:57.564421 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:31:57.564427 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:31:57.564430 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:31:57.564525 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 03:31:57.564629 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
I20260927 03:31:57.565299 20421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35011
I20260927 03:31:57.565403 20421 heartbeater.cc:464] Registering TS with master...
I20260927 03:31:57.565593 20421 heartbeater.cc:511] Master 127.0.98.62:35011 requested a full tablet report, sending...
I20260927 03:31:57.565986 20102 ts_manager.cc:194] Registered new tserver with Master: 2f8ded2ae59c40ebaa78644bcdc5c757 (127.0.98.2:45951)
I20260927 03:31:57.566537 20102 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:46833
W20260927 03:31:57.643392 20424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:57.643617 20424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:57.643640 20424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:57.644845 20424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:57.644892 20424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:57.644917 20424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 03:31:57.646275 20424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:57.646476 20424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:57.646782 20424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:31:57.651767 20432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:57.651706 20430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:57.651718 20429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:57.652009 20424 server_base.cc:1034] running on GCE node
I20260927 03:31:57.652182 20424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:57.652369 20424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:57.653499 20424 hybrid_clock.cc:648] HybridClock initialized: now 1790479917653485 us; error 32 us; skew 500 ppm
I20260927 03:31:57.654680 20424 webserver.cc:460] Webserver started at http://127.0.98.3:44787/ using document root <none> and password file <none>
I20260927 03:31:57.654903 20424 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:57.654949 20424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:57.655115 20424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:31:57.656220 20424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-2/data/instance:
uuid: "b674d19eafab40b394344e625fa9860b"
format_stamp: "Formatted at 2026-09-27 03:31:57 on dist-test-slave-j0hf"
server_key: "82f20828ad835623d52374c32f0ba971"
server_key_iv: "ce16c5c07f282e2616803094705e3370"
server_key_version: "encryptionkey@0"
I20260927 03:31:57.656562 20424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b674d19eafab40b394344e625fa9860b"
format_stamp: "Formatted at 2026-09-27 03:31:57 on dist-test-slave-j0hf"
server_key: "82f20828ad835623d52374c32f0ba971"
server_key_iv: "ce16c5c07f282e2616803094705e3370"
server_key_version: "encryptionkey@0"
I20260927 03:31:57.657795 20424 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:31:57.658560 20438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:57.658748 20424 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:31:57.658847 20424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-2/wal
uuid: "b674d19eafab40b394344e625fa9860b"
format_stamp: "Formatted at 2026-09-27 03:31:57 on dist-test-slave-j0hf"
server_key: "82f20828ad835623d52374c32f0ba971"
server_key_iv: "ce16c5c07f282e2616803094705e3370"
server_key_version: "encryptionkey@0"
I20260927 03:31:57.658916 20424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:57.678352 20424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:57.678689 20424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:57.678840 20424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:57.679081 20424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:31:57.679414 20424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:31:57.679526 20424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:57.679562 20424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:31:57.679591 20424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:57.685321 20424 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33531
I20260927 03:31:57.685417 20551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33531 every 8 connection(s)
I20260927 03:31:57.685743 20424 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-2/data/info.pb
I20260927 03:31:57.687325 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 20424
I20260927 03:31:57.687476 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-2/wal/instance
I20260927 03:31:57.687677 392 external_mini_cluster.cc:1428] Setting key a8d8220287a97c09ff095ee90521835b
I20260927 03:31:57.691080 20141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:57.686529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48965 (local address 127.0.98.62:35011)
0927 03:31:57.686594 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:57.686600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:57.686829 (+ 229us) server_negotiation.cc:407] Connection header received
0927 03:31:57.687367 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:57.687369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:57.687382 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:57.687454 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:57.688310 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.688800 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.689776 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.689967 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.690053 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:57.690819 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:57.690821 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:57.690821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:31:57.690827 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:57.690841 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:31:57.690846 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:57.690896 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:57.690968 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 03:31:57.691003 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 03:31:57.691288 20554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:57.686515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35011 (local address 127.0.98.3:48965)
0927 03:31:57.686712 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:57.686745 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:57.687251 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:57.687535 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:57.687558 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:57.687568 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:31:57.688219 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:57.688225 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.688887 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.688888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:57.689680 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:57.689682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.690061 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.690062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:57.690199 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:57.690686 (+ 487us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:31:57.690686 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:31:57.690715 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:31:57.690737 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:31:57.690901 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:31:57.690905 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:31:57.690908 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:31:57.690973 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 03:31:57.691070 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260927 03:31:57.691776 20552 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35011
I20260927 03:31:57.691851 20552 heartbeater.cc:464] Registering TS with master...
I20260927 03:31:57.692044 20552 heartbeater.cc:511] Master 127.0.98.62:35011 requested a full tablet report, sending...
I20260927 03:31:57.692410 20102 ts_manager.cc:194] Registered new tserver with Master: b674d19eafab40b394344e625fa9860b (127.0.98.3:33531)
I20260927 03:31:57.692778 20102 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:48965
I20260927 03:31:57.699347 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:31:57.703375 20141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:57.700752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60828 (local address 127.0.98.62:35011)
0927 03:31:57.700818 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:57.700819 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:57.700936 (+ 117us) server_negotiation.cc:407] Connection header received
0927 03:31:57.700977 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:57.700978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:57.700986 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:57.701044 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:57.701198 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.701664 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.702380 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.702551 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.702600 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:57.703196 (+ 596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:57.703197 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:57.703197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:31:57.703201 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:57.703213 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:31:57.703217 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:57.703253 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:57.703300 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 03:31:57.703320 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 03:31:57.705049 20102 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:31:57.706631 20102 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:31:57.712873 20561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:57.708729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45951 (local address 127.0.0.1:48498)
0927 03:31:57.708911 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:57.708953 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:57.709029 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:57.709416 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:57.709419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:57.709421 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:57.709490 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:57.709492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.710832 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.710833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:57.711883 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:57.711885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.712583 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.712584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:57.712660 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:57.712675 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 03:31:57.712749 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 03:31:57.712797 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":57,"threads_started":1}
I20260927 03:31:57.712890 20565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:57.709061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48498 (local address 127.0.98.2:45951)
0927 03:31:57.709207 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:57.709230 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:57.709243 (+ 13us) server_negotiation.cc:407] Connection header received
0927 03:31:57.709286 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:57.709289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:57.709306 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:57.709411 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:57.709573 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.710746 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.711958 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.712495 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.712567 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:57.712659 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:57.712770 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 03:31:57.712807 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260927 03:31:57.713435 20563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:57.708759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33531 (local address 127.0.0.1:49922)
0927 03:31:57.709209 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:57.709218 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:57.709261 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:57.709549 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:57.709551 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:57.709552 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:57.709583 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:57.709584 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.711473 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.711474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:57.712328 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:57.712330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.713234 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.713235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:57.713298 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:57.713303 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:31:57.713333 (+ 30us) client_negotiation.cc:240] Negotiation successful
0927 03:31:57.713369 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":97,"threads_started":1}
I20260927 03:31:57.713817 20562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:57.708842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49922 (local address 127.0.98.3:33531)
0927 03:31:57.709396 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:57.709400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:57.709409 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:31:57.709444 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:57.709446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:57.709459 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:57.709543 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:57.709662 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.710744 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.712404 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.713130 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.713578 (+ 448us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:57.713670 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:57.713697 (+ 27us) server_negotiation.cc:299] Negotiation successful
0927 03:31:57.713736 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":89,"threads_started":1}
I20260927 03:31:57.713662 20355 tablet_service.cc:1467] Processing CreateTablet for tablet a7d0d42bf50c49fab97d26cc3ee21b67 (DEFAULT_TABLE table=test-table [id=78c886b668554ae1995618c6802ab677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:31:57.714077 20355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d0d42bf50c49fab97d26cc3ee21b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:31:57.714463 20486 tablet_service.cc:1467] Processing CreateTablet for tablet a7d0d42bf50c49fab97d26cc3ee21b67 (DEFAULT_TABLE table=test-table [id=78c886b668554ae1995618c6802ab677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:31:57.714820 20486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d0d42bf50c49fab97d26cc3ee21b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:31:57.715123 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:57.708931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33043 (local address 127.0.0.1:58222)
0927 03:31:57.709230 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:57.709241 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:57.709313 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:57.709768 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:57.709769 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:57.709771 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:57.709811 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:57.709812 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.712398 (+ 2586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.712398 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:57.713443 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:57.713446 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.714877 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.714878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:57.714944 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:57.714949 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:31:57.715013 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 03:31:57.715053 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":61,"threads_started":1}
I20260927 03:31:57.715399 20564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:57.709031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58222 (local address 127.0.98.1:33043)
0927 03:31:57.709184 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:57.709189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:57.709302 (+ 113us) server_negotiation.cc:407] Connection header received
0927 03:31:57.709391 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:57.709394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:57.709620 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:57.709705 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:57.709890 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.710748 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.713719 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.714686 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.714756 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:57.714839 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:57.715277 (+ 438us) server_negotiation.cc:299] Negotiation successful
0927 03:31:57.715322 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260927 03:31:57.716054 20224 tablet_service.cc:1467] Processing CreateTablet for tablet a7d0d42bf50c49fab97d26cc3ee21b67 (DEFAULT_TABLE table=test-table [id=78c886b668554ae1995618c6802ab677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:31:57.716361 20224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d0d42bf50c49fab97d26cc3ee21b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:31:57.717402 20566 tablet_bootstrap.cc:492] T a7d0d42bf50c49fab97d26cc3ee21b67 P 2f8ded2ae59c40ebaa78644bcdc5c757: Bootstrap starting.
I20260927 03:31:57.717706 20567 tablet_bootstrap.cc:492] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b: Bootstrap starting.
I20260927 03:31:57.718196 20566 tablet_bootstrap.cc:654] T a7d0d42bf50c49fab97d26cc3ee21b67 P 2f8ded2ae59c40ebaa78644bcdc5c757: Neither blocks nor log segments found. Creating new log.
I20260927 03:31:57.718348 20567 tablet_bootstrap.cc:654] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b: Neither blocks nor log segments found. Creating new log.
I20260927 03:31:57.718536 20566 log.cc:824] T a7d0d42bf50c49fab97d26cc3ee21b67 P 2f8ded2ae59c40ebaa78644bcdc5c757: Log is configured to *not* fsync() on all Append() calls
I20260927 03:31:57.718662 20567 log.cc:824] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b: Log is configured to *not* fsync() on all Append() calls
I20260927 03:31:57.719340 20567 tablet_bootstrap.cc:492] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b: No bootstrap required, opened a new log
I20260927 03:31:57.719347 20566 tablet_bootstrap.cc:492] T a7d0d42bf50c49fab97d26cc3ee21b67 P 2f8ded2ae59c40ebaa78644bcdc5c757: No bootstrap required, opened a new log
I20260927 03:31:57.719420 20567 ts_tablet_manager.cc:1397] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:31:57.719456 20566 ts_tablet_manager.cc:1397] T a7d0d42bf50c49fab97d26cc3ee21b67 P 2f8ded2ae59c40ebaa78644bcdc5c757: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260927 03:31:57.719740 20570 tablet_bootstrap.cc:492] T a7d0d42bf50c49fab97d26cc3ee21b67 P 460444f334f34b11af91ddef6590f344: Bootstrap starting.
I20260927 03:31:57.720502 20570 tablet_bootstrap.cc:654] T a7d0d42bf50c49fab97d26cc3ee21b67 P 460444f334f34b11af91ddef6590f344: Neither blocks nor log segments found. Creating new log.
I20260927 03:31:57.720806 20570 log.cc:824] T a7d0d42bf50c49fab97d26cc3ee21b67 P 460444f334f34b11af91ddef6590f344: Log is configured to *not* fsync() on all Append() calls
I20260927 03:31:57.721560 20570 tablet_bootstrap.cc:492] T a7d0d42bf50c49fab97d26cc3ee21b67 P 460444f334f34b11af91ddef6590f344: No bootstrap required, opened a new log
I20260927 03:31:57.721632 20570 ts_tablet_manager.cc:1397] T a7d0d42bf50c49fab97d26cc3ee21b67 P 460444f334f34b11af91ddef6590f344: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:31:57.721833 20566 raft_consensus.cc:348] T a7d0d42bf50c49fab97d26cc3ee21b67 P 2f8ded2ae59c40ebaa78644bcdc5c757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8ded2ae59c40ebaa78644bcdc5c757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45951 } } peers { permanent_uuid: "460444f334f34b11af91ddef6590f344" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "b674d19eafab40b394344e625fa9860b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33531 } }
I20260927 03:31:57.722016 20566 raft_consensus.cc:374] T a7d0d42bf50c49fab97d26cc3ee21b67 P 2f8ded2ae59c40ebaa78644bcdc5c757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:31:57.722060 20566 raft_consensus.cc:729] T a7d0d42bf50c49fab97d26cc3ee21b67 P 2f8ded2ae59c40ebaa78644bcdc5c757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f8ded2ae59c40ebaa78644bcdc5c757, State: Initialized, Role: FOLLOWER
I20260927 03:31:57.722054 20567 raft_consensus.cc:348] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8ded2ae59c40ebaa78644bcdc5c757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45951 } } peers { permanent_uuid: "460444f334f34b11af91ddef6590f344" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "b674d19eafab40b394344e625fa9860b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33531 } }
I20260927 03:31:57.722219 20567 raft_consensus.cc:374] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:31:57.722203 20566 consensus_queue.cc:260] T a7d0d42bf50c49fab97d26cc3ee21b67 P 2f8ded2ae59c40ebaa78644bcdc5c757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8ded2ae59c40ebaa78644bcdc5c757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45951 } } peers { permanent_uuid: "460444f334f34b11af91ddef6590f344" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "b674d19eafab40b394344e625fa9860b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33531 } }
I20260927 03:31:57.722262 20567 raft_consensus.cc:729] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b674d19eafab40b394344e625fa9860b, State: Initialized, Role: FOLLOWER
I20260927 03:31:57.722404 20567 consensus_queue.cc:260] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8ded2ae59c40ebaa78644bcdc5c757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45951 } } peers { permanent_uuid: "460444f334f34b11af91ddef6590f344" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "b674d19eafab40b394344e625fa9860b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33531 } }
I20260927 03:31:57.722515 20421 heartbeater.cc:503] Master 127.0.98.62:35011 was elected leader, sending a full tablet report...
I20260927 03:31:57.722695 20567 ts_tablet_manager.cc:1428] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 03:31:57.722707 20552 heartbeater.cc:503] Master 127.0.98.62:35011 was elected leader, sending a full tablet report...
I20260927 03:31:57.722890 20566 ts_tablet_manager.cc:1428] T a7d0d42bf50c49fab97d26cc3ee21b67 P 2f8ded2ae59c40ebaa78644bcdc5c757: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 03:31:57.723774 20573 raft_consensus.cc:482] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:31:57.723861 20573 raft_consensus.cc:504] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8ded2ae59c40ebaa78644bcdc5c757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45951 } } peers { permanent_uuid: "460444f334f34b11af91ddef6590f344" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "b674d19eafab40b394344e625fa9860b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33531 } }
I20260927 03:31:57.724166 20573 leader_election.cc:288] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f8ded2ae59c40ebaa78644bcdc5c757 (127.0.98.2:45951), 460444f334f34b11af91ddef6590f344 (127.0.98.1:33043)
I20260927 03:31:57.724404 20570 raft_consensus.cc:348] T a7d0d42bf50c49fab97d26cc3ee21b67 P 460444f334f34b11af91ddef6590f344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8ded2ae59c40ebaa78644bcdc5c757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45951 } } peers { permanent_uuid: "460444f334f34b11af91ddef6590f344" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "b674d19eafab40b394344e625fa9860b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33531 } }
I20260927 03:31:57.724570 20570 raft_consensus.cc:374] T a7d0d42bf50c49fab97d26cc3ee21b67 P 460444f334f34b11af91ddef6590f344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:31:57.724619 20570 raft_consensus.cc:729] T a7d0d42bf50c49fab97d26cc3ee21b67 P 460444f334f34b11af91ddef6590f344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 460444f334f34b11af91ddef6590f344, State: Initialized, Role: FOLLOWER
I20260927 03:31:57.724726 20570 consensus_queue.cc:260] T a7d0d42bf50c49fab97d26cc3ee21b67 P 460444f334f34b11af91ddef6590f344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8ded2ae59c40ebaa78644bcdc5c757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45951 } } peers { permanent_uuid: "460444f334f34b11af91ddef6590f344" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "b674d19eafab40b394344e625fa9860b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33531 } }
I20260927 03:31:57.725009 20570 ts_tablet_manager.cc:1428] T a7d0d42bf50c49fab97d26cc3ee21b67 P 460444f334f34b11af91ddef6590f344: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260927 03:31:57.725133 20290 heartbeater.cc:503] Master 127.0.98.62:35011 was elected leader, sending a full tablet report...
I20260927 03:31:57.727161 20565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:57.724465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43251 (local address 127.0.98.2:45951)
0927 03:31:57.724525 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:57.724527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:57.724535 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:31:57.724559 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:57.724560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:57.724569 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:57.724627 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:57.724804 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.725565 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.726467 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.726831 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.726911 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:57.726920 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:57.727067 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 03:31:57.727101 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 03:31:57.727164 20554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:57.724298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45951 (local address 127.0.98.3:43251)
0927 03:31:57.724335 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:57.724347 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:57.724385 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:57.724677 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:57.724678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:57.724680 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:57.724741 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:57.724743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.725632 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.725633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:57.726377 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:57.726379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.726910 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.726911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:57.726978 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:57.726982 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:31:57.727058 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 03:31:57.727100 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260927 03:31:57.727461 20564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:57.724445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41179 (local address 127.0.98.1:33043)
0927 03:31:57.724491 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:57.724493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:57.724603 (+ 110us) server_negotiation.cc:407] Connection header received
0927 03:31:57.724697 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:57.724698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:57.724728 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:57.724756 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:31:57.724947 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.725702 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.726688 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.727048 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.727130 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:57.727138 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:57.727348 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 03:31:57.727382 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:31:57.727471 20375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d0d42bf50c49fab97d26cc3ee21b67" candidate_uuid: "b674d19eafab40b394344e625fa9860b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f8ded2ae59c40ebaa78644bcdc5c757" is_pre_election: true
I20260927 03:31:57.727622 20375 raft_consensus.cc:2454] T a7d0d42bf50c49fab97d26cc3ee21b67 P 2f8ded2ae59c40ebaa78644bcdc5c757 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b674d19eafab40b394344e625fa9860b in term 0.
I20260927 03:31:57.727677 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:57.724409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33043 (local address 127.0.98.3:41179)
0927 03:31:57.724530 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:57.724539 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:31:57.724627 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:31:57.724814 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:31:57.724815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:31:57.724817 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:31:57.724875 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:57.724877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.725770 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.725770 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:57.726603 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:31:57.726606 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.727135 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.727136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:31:57.727200 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:57.727204 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:31:57.727545 (+ 341us) client_negotiation.cc:240] Negotiation successful
0927 03:31:57.727591 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
I20260927 03:31:57.727856 20440 leader_election.cc:302] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f8ded2ae59c40ebaa78644bcdc5c757, b674d19eafab40b394344e625fa9860b; no voters:
I20260927 03:31:57.727936 20244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d0d42bf50c49fab97d26cc3ee21b67" candidate_uuid: "b674d19eafab40b394344e625fa9860b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "460444f334f34b11af91ddef6590f344" is_pre_election: true
I20260927 03:31:57.727977 20573 raft_consensus.cc:2781] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:31:57.728034 20573 raft_consensus.cc:482] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:31:57.728042 20244 raft_consensus.cc:2454] T a7d0d42bf50c49fab97d26cc3ee21b67 P 460444f334f34b11af91ddef6590f344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b674d19eafab40b394344e625fa9860b in term 0.
I20260927 03:31:57.728061 20573 raft_consensus.cc:3037] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:31:57.728710 20573 raft_consensus.cc:504] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8ded2ae59c40ebaa78644bcdc5c757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45951 } } peers { permanent_uuid: "460444f334f34b11af91ddef6590f344" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "b674d19eafab40b394344e625fa9860b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33531 } }
I20260927 03:31:57.728832 20573 leader_election.cc:288] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b [CANDIDATE]: Term 1 election: Requested vote from peers 2f8ded2ae59c40ebaa78644bcdc5c757 (127.0.98.2:45951), 460444f334f34b11af91ddef6590f344 (127.0.98.1:33043)
I20260927 03:31:57.728983 20375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d0d42bf50c49fab97d26cc3ee21b67" candidate_uuid: "b674d19eafab40b394344e625fa9860b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f8ded2ae59c40ebaa78644bcdc5c757"
I20260927 03:31:57.728988 20244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d0d42bf50c49fab97d26cc3ee21b67" candidate_uuid: "b674d19eafab40b394344e625fa9860b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "460444f334f34b11af91ddef6590f344"
I20260927 03:31:57.729048 20375 raft_consensus.cc:3037] T a7d0d42bf50c49fab97d26cc3ee21b67 P 2f8ded2ae59c40ebaa78644bcdc5c757 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:31:57.729064 20244 raft_consensus.cc:3037] T a7d0d42bf50c49fab97d26cc3ee21b67 P 460444f334f34b11af91ddef6590f344 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:31:57.729687 20375 raft_consensus.cc:2454] T a7d0d42bf50c49fab97d26cc3ee21b67 P 2f8ded2ae59c40ebaa78644bcdc5c757 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b674d19eafab40b394344e625fa9860b in term 1.
I20260927 03:31:57.729760 20244 raft_consensus.cc:2454] T a7d0d42bf50c49fab97d26cc3ee21b67 P 460444f334f34b11af91ddef6590f344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b674d19eafab40b394344e625fa9860b in term 1.
I20260927 03:31:57.729836 20440 leader_election.cc:302] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f8ded2ae59c40ebaa78644bcdc5c757, b674d19eafab40b394344e625fa9860b; no voters:
I20260927 03:31:57.729907 20573 raft_consensus.cc:2781] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:31:57.730046 20573 raft_consensus.cc:686] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b [term 1 LEADER]: Becoming Leader. State: Replica: b674d19eafab40b394344e625fa9860b, State: Running, Role: LEADER
I20260927 03:31:57.730142 20573 consensus_queue.cc:237] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8ded2ae59c40ebaa78644bcdc5c757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45951 } } peers { permanent_uuid: "460444f334f34b11af91ddef6590f344" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } } peers { permanent_uuid: "b674d19eafab40b394344e625fa9860b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33531 } }
I20260927 03:31:57.730888 20102 catalog_manager.cc:5495] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b reported cstate change: term changed from 0 to 1, leader changed from <none> to b674d19eafab40b394344e625fa9860b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b674d19eafab40b394344e625fa9860b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f8ded2ae59c40ebaa78644bcdc5c757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "460444f334f34b11af91ddef6590f344" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b674d19eafab40b394344e625fa9860b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33531 } health_report { overall_health: HEALTHY } } }
I20260927 03:31:57.741528 20562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:57.739273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49930 (local address 127.0.98.3:33531)
0927 03:31:57.739337 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:57.739339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:57.739347 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:31:57.739404 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:57.739405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:57.739456 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:57.739536 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:31:57.739627 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.740090 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.740792 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:57.740977 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:57.741054 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:57.741171 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:31:57.741272 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:31:57.741334 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:57.741365 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 03:31:57.741381 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 03:31:57.743368 20244 raft_consensus.cc:1261] T a7d0d42bf50c49fab97d26cc3ee21b67 P 460444f334f34b11af91ddef6590f344 [term 1 FOLLOWER]: Refusing update from remote peer b674d19eafab40b394344e625fa9860b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:31:57.743376 20375 raft_consensus.cc:1261] T a7d0d42bf50c49fab97d26cc3ee21b67 P 2f8ded2ae59c40ebaa78644bcdc5c757 [term 1 FOLLOWER]: Refusing update from remote peer b674d19eafab40b394344e625fa9860b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:31:57.743788 20573 consensus_queue.cc:1035] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f8ded2ae59c40ebaa78644bcdc5c757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:31:57.743940 20576 consensus_queue.cc:1035] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b [LEADER]: Connected to new peer: Peer: permanent_uuid: "460444f334f34b11af91ddef6590f344" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:31:57.746611 20581 mvcc.cc:205] Tried to move back new op lower bound from 7333805743074021376 to 7333805743023353856. Current Snapshot: MvccSnapshot[applied={T|T < 7333805743074021376}]
I20260927 03:31:57.747162 20580 mvcc.cc:205] Tried to move back new op lower bound from 7333805743074021376 to 7333805743023353856. Current Snapshot: MvccSnapshot[applied={T|T < 7333805743074021376}]
I20260927 03:31:57.748868 20101 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:31:57.748965 20101 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:31:57.749734 20101 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f57cbec513e64c6698cbdcc071015779: Sending DeleteTablet for 3 replicas of tablet a7d0d42bf50c49fab97d26cc3ee21b67
I20260927 03:31:57.750146 20224 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d0d42bf50c49fab97d26cc3ee21b67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:31:57 UTC) from {username='slave'} at 127.0.0.1:58222
I20260927 03:31:57.750172 20486 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d0d42bf50c49fab97d26cc3ee21b67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:31:57 UTC) from {username='slave'} at 127.0.0.1:49922
I20260927 03:31:57.750365 20355 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d0d42bf50c49fab97d26cc3ee21b67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:31:57 UTC) from {username='slave'} at 127.0.0.1:48498
I20260927 03:31:57.750593 20590 tablet_replica.cc:330] T a7d0d42bf50c49fab97d26cc3ee21b67 P 2f8ded2ae59c40ebaa78644bcdc5c757: stopping tablet replica
I20260927 03:31:57.750586 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 20162
I20260927 03:31:57.750594 20588 tablet_replica.cc:330] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b: stopping tablet replica
I20260927 03:31:57.751452 20590 raft_consensus.cc:2229] T a7d0d42bf50c49fab97d26cc3ee21b67 P 2f8ded2ae59c40ebaa78644bcdc5c757 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:31:57.751659 20590 raft_consensus.cc:2258] T a7d0d42bf50c49fab97d26cc3ee21b67 P 2f8ded2ae59c40ebaa78644bcdc5c757 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:31:57.752310 20590 ts_tablet_manager.cc:1905] T a7d0d42bf50c49fab97d26cc3ee21b67 P 2f8ded2ae59c40ebaa78644bcdc5c757: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:31:57.754594 20590 ts_tablet_manager.cc:1918] T a7d0d42bf50c49fab97d26cc3ee21b67 P 2f8ded2ae59c40ebaa78644bcdc5c757: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:31:57.754675 20590 log.cc:1196] T a7d0d42bf50c49fab97d26cc3ee21b67 P 2f8ded2ae59c40ebaa78644bcdc5c757: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-1/wal/wals/a7d0d42bf50c49fab97d26cc3ee21b67
I20260927 03:31:57.754946 20590 ts_tablet_manager.cc:1939] T a7d0d42bf50c49fab97d26cc3ee21b67 P 2f8ded2ae59c40ebaa78644bcdc5c757: Deleting consensus metadata
I20260927 03:31:57.755328 20087 catalog_manager.cc:4841] TS 2f8ded2ae59c40ebaa78644bcdc5c757 (127.0.98.2:45951): tablet a7d0d42bf50c49fab97d26cc3ee21b67 (table test-table [id=78c886b668554ae1995618c6802ab677]) successfully deleted
W20260927 03:31:57.757205 20087 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33043 (error 108)
I20260927 03:31:57.757328 20588 raft_consensus.cc:2229] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:31:57.757462 20588 raft_consensus.cc:2258] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:31:57.757663 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 20293
I20260927 03:31:57.757810 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:57.757680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33043 (local address 127.0.0.1:58224)
0927 03:31:57.757733 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 03:31:57.757772 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 03:31:57.758325 20087 catalog_manager.cc:4584] TS 460444f334f34b11af91ddef6590f344 (127.0.98.1:33043): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a7d0d42bf50c49fab97d26cc3ee21b67: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33043: connect: Connection refused (error 111)
I20260927 03:31:57.759068 20588 ts_tablet_manager.cc:1905] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:31:57.760478 20588 ts_tablet_manager.cc:1918] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:31:57.760571 20588 log.cc:1196] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b: Deleting WAL directory at /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790479675865103-392-0/minicluster-data/ts-2/wal/wals/a7d0d42bf50c49fab97d26cc3ee21b67
I20260927 03:31:57.760800 20588 ts_tablet_manager.cc:1939] T a7d0d42bf50c49fab97d26cc3ee21b67 P b674d19eafab40b394344e625fa9860b: Deleting consensus metadata
I20260927 03:31:57.761278 20089 catalog_manager.cc:4841] TS b674d19eafab40b394344e625fa9860b (127.0.98.3:33531): tablet a7d0d42bf50c49fab97d26cc3ee21b67 (table test-table [id=78c886b668554ae1995618c6802ab677]) successfully deleted
I20260927 03:31:57.765012 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 20424
I20260927 03:31:57.770264 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 20071
2026-09-27T03:31:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (649 ms)
[----------] 6 tests from EncryptionPolicyTest (8300 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:31:57 dist-test-slave-j0hf krb5kdc[20594](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:31:57 dist-test-slave-j0hf krb5kdc[20594](info): set up 2 sockets
Sep 27 03:31:57 dist-test-slave-j0hf krb5kdc[20594](info): commencing operation
krb5kdc: starting...
W20260927 03:31:58.811405 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:31:58 dist-test-slave-j0hf krb5kdc[20594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:31:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:31:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:31:58.926293 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40115
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:31:59.006405 20607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:31:59.006642 20607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:31:59.006678 20607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:31:59.007848 20607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:31:59.007910 20607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:31:59.007932 20607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:31:59.007953 20607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:31:59.007997 20607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:31:59.009449 20607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40115
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:31:59.009709 20607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:31:59.009939 20607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:31:59.014659 20612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:59.014680 20613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:31:59.014659 20615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:31:59.014993 20607 server_base.cc:1034] running on GCE node
I20260927 03:31:59.015235 20607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:31:59.015574 20607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:31:59.016739 20607 hybrid_clock.cc:648] HybridClock initialized: now 1790479919016721 us; error 29 us; skew 500 ppm
Sep 27 03:31:59 dist-test-slave-j0hf krb5kdc[20594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:31:59.019171 20607 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:31:59.019668 20607 webserver.cc:460] Webserver started at http://127.0.0.1:46329/ using document root <none> and password file <none>
I20260927 03:31:59.019914 20607 fs_manager.cc:359] Metadata directory not provided
I20260927 03:31:59.019980 20607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:31:59.020124 20607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:31:59.021232 20607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "39dea4b6bef043ccaee8ca9467a55a91"
format_stamp: "Formatted at 2026-09-27 03:31:59 on dist-test-slave-j0hf"
server_key: "49dcf74226f61c5768998c3c546744c6"
server_key_iv: "e5d4602cac067a054fc8cc632d21fa34"
server_key_version: "encryptionkey@0"
I20260927 03:31:59.021587 20607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "39dea4b6bef043ccaee8ca9467a55a91"
format_stamp: "Formatted at 2026-09-27 03:31:59 on dist-test-slave-j0hf"
server_key: "49dcf74226f61c5768998c3c546744c6"
server_key_iv: "e5d4602cac067a054fc8cc632d21fa34"
server_key_version: "encryptionkey@0"
I20260927 03:31:59.022897 20607 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:31:59.023607 20622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:31:59.023774 20607 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:31:59.023902 20607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "39dea4b6bef043ccaee8ca9467a55a91"
format_stamp: "Formatted at 2026-09-27 03:31:59 on dist-test-slave-j0hf"
server_key: "49dcf74226f61c5768998c3c546744c6"
server_key_iv: "e5d4602cac067a054fc8cc632d21fa34"
server_key_version: "encryptionkey@0"
I20260927 03:31:59.023979 20607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:31:59.058808 20607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:31:59.061122 20607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:31:59.061292 20607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:31:59.065889 20607 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40115
I20260927 03:31:59.065936 20674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40115 every 8 connection(s)
I20260927 03:31:59.066332 20607 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:31:59.067195 20675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:31:59.069649 20675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39dea4b6bef043ccaee8ca9467a55a91: Bootstrap starting.
I20260927 03:31:59.070263 20675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39dea4b6bef043ccaee8ca9467a55a91: Neither blocks nor log segments found. Creating new log.
I20260927 03:31:59.070590 20675 log.cc:824] T 00000000000000000000000000000000 P 39dea4b6bef043ccaee8ca9467a55a91: Log is configured to *not* fsync() on all Append() calls
I20260927 03:31:59.071398 20675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39dea4b6bef043ccaee8ca9467a55a91: No bootstrap required, opened a new log
I20260927 03:31:59.071385 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 20607
I20260927 03:31:59.071534 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:31:59.071743 392 external_mini_cluster.cc:1428] Setting key 63f6dd680cdc367d42b3a6167e4d6eec
I20260927 03:31:59.073429 20675 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39dea4b6bef043ccaee8ca9467a55a91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39dea4b6bef043ccaee8ca9467a55a91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40115 } }
I20260927 03:31:59.073580 20675 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39dea4b6bef043ccaee8ca9467a55a91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:31:59.073602 20675 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39dea4b6bef043ccaee8ca9467a55a91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39dea4b6bef043ccaee8ca9467a55a91, State: Initialized, Role: FOLLOWER
I20260927 03:31:59.073729 20675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39dea4b6bef043ccaee8ca9467a55a91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39dea4b6bef043ccaee8ca9467a55a91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40115 } }
I20260927 03:31:59.073812 20675 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39dea4b6bef043ccaee8ca9467a55a91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:31:59.073848 20675 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39dea4b6bef043ccaee8ca9467a55a91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:31:59.073895 20675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39dea4b6bef043ccaee8ca9467a55a91 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:31:59.075009 20675 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39dea4b6bef043ccaee8ca9467a55a91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39dea4b6bef043ccaee8ca9467a55a91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40115 } }
I20260927 03:31:59.075098 20675 leader_election.cc:302] T 00000000000000000000000000000000 P 39dea4b6bef043ccaee8ca9467a55a91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39dea4b6bef043ccaee8ca9467a55a91; no voters:
I20260927 03:31:59.075270 20675 leader_election.cc:288] T 00000000000000000000000000000000 P 39dea4b6bef043ccaee8ca9467a55a91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:31:59.075321 20680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39dea4b6bef043ccaee8ca9467a55a91 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:31:59.075487 20680 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39dea4b6bef043ccaee8ca9467a55a91 [term 1 LEADER]: Becoming Leader. State: Replica: 39dea4b6bef043ccaee8ca9467a55a91, State: Running, Role: LEADER
I20260927 03:31:59.075555 20675 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39dea4b6bef043ccaee8ca9467a55a91 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:31:59.075574 20680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39dea4b6bef043ccaee8ca9467a55a91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39dea4b6bef043ccaee8ca9467a55a91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40115 } }
I20260927 03:31:59.075909 20680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39dea4b6bef043ccaee8ca9467a55a91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39dea4b6bef043ccaee8ca9467a55a91. Latest consensus state: current_term: 1 leader_uuid: "39dea4b6bef043ccaee8ca9467a55a91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39dea4b6bef043ccaee8ca9467a55a91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40115 } } }
I20260927 03:31:59.075999 20680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39dea4b6bef043ccaee8ca9467a55a91 [sys.catalog]: This master's current role is: LEADER
I20260927 03:31:59.075922 20681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39dea4b6bef043ccaee8ca9467a55a91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39dea4b6bef043ccaee8ca9467a55a91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39dea4b6bef043ccaee8ca9467a55a91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40115 } } }
I20260927 03:31:59.076113 20681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39dea4b6bef043ccaee8ca9467a55a91 [sys.catalog]: This master's current role is: LEADER
I20260927 03:31:59.076587 20686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:31:59.077112 20686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:31:59.078764 20686 catalog_manager.cc:1347] Generated new cluster ID: b0b3e340883541f3999e70a76f6d48c5
I20260927 03:31:59.078820 20686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:31:59 dist-test-slave-j0hf krb5kdc[20594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 03:31:59.084854 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:59.072312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47961 (local address 127.0.0.1:40115)
0927 03:31:59.072499 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:59.072509 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:59.072544 (+ 35us) server_negotiation.cc:407] Connection header received
0927 03:31:59.073438 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:59.073447 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:59.073511 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:59.073701 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:59.074579 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:59.075845 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:59.076957 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:59.077615 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:59.077695 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:59.080716 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:59.080720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:59.080720 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:31:59.080732 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:59.082716 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:59.083196 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:59.083199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:59.083200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:59.083329 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:59.083615 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:59.083616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:59.083617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:59.083930 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:59.084114 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:59.084547 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 03:31:59.084657 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
I20260927 03:31:59.085839 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:31:59.091995 20686 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:31:59.092593 20686 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:31:59.098843 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:31:59.087532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60497 (local address 127.0.0.1:40115)
0927 03:31:59.087566 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:31:59.087568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:31:59.091650 (+ 4082us) server_negotiation.cc:407] Connection header received
0927 03:31:59.091684 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:31:59.091685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:31:59.091696 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:31:59.091758 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:31:59.092436 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:59.093140 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:59.094155 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:31:59.094467 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:31:59.094534 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:31:59.096032 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:31:59.096034 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:31:59.096034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:31:59.096041 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:31:59.097629 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:59.098047 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:59.098048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:59.098049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:59.098105 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:31:59.098329 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:31:59.098330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:31:59.098330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:31:59.098445 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:31:59.098544 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 03:31:59.098686 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 03:31:59.098748 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 03:31:59.102917 20686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39dea4b6bef043ccaee8ca9467a55a91: Generated new TSK 0
I20260927 03:31:59.103102 20686 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:31:59.107825 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 20607
W20260927 03:31:59.111925 20592 connection.cc:535] client connection to 127.0.0.1:40115 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40115): Connection reset by peer (error 104)
2026-09-27T03:31:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:31:59 dist-test-slave-j0hf krb5kdc[20706](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:31:59 dist-test-slave-j0hf krb5kdc[20706](info): set up 2 sockets
Sep 27 03:31:59 dist-test-slave-j0hf krb5kdc[20706](info): commencing operation
krb5kdc: starting...
W20260927 03:32:00.139603 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:32:00 dist-test-slave-j0hf krb5kdc[20706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:32:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:32:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:32:00.256038 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45731
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:32:00.337978 20719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:32:00.338193 20719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:32:00.338217 20719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:32:00.339329 20719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:32:00.339377 20719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:32:00.339392 20719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:32:00.339404 20719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:32:00.339416 20719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:32:00.340977 20719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45731
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:32:00.341176 20719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:32:00.341344 20719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:32:00.346092 20725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:00.346087 20724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:00.346087 20727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:32:00.346242 20719 server_base.cc:1034] running on GCE node
I20260927 03:32:00.346652 20719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:32:00.346971 20719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:32:00.348121 20719 hybrid_clock.cc:648] HybridClock initialized: now 1790479920348106 us; error 42 us; skew 500 ppm
Sep 27 03:32:00 dist-test-slave-j0hf krb5kdc[20706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:32:00.357213 20719 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:32:00.357712 20719 webserver.cc:460] Webserver started at http://127.0.0.1:43137/ using document root <none> and password file <none>
I20260927 03:32:00.357932 20719 fs_manager.cc:359] Metadata directory not provided
I20260927 03:32:00.357998 20719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:32:00.358147 20719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:32:00.359229 20719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "0a9260143cd74ac3bd4228547dbc4ecb"
format_stamp: "Formatted at 2026-09-27 03:32:00 on dist-test-slave-j0hf"
server_key: "f99f0bcc86190eaa443ede1daacb84c5"
server_key_iv: "3c56a3a31960da9fecd5a84af7a5e3fa"
server_key_version: "encryptionkey@0"
I20260927 03:32:00.359614 20719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "0a9260143cd74ac3bd4228547dbc4ecb"
format_stamp: "Formatted at 2026-09-27 03:32:00 on dist-test-slave-j0hf"
server_key: "f99f0bcc86190eaa443ede1daacb84c5"
server_key_iv: "3c56a3a31960da9fecd5a84af7a5e3fa"
server_key_version: "encryptionkey@0"
I20260927 03:32:00.360984 20719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:32:00.361676 20734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:32:00.361833 20719 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:32:00.361931 20719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "0a9260143cd74ac3bd4228547dbc4ecb"
format_stamp: "Formatted at 2026-09-27 03:32:00 on dist-test-slave-j0hf"
server_key: "f99f0bcc86190eaa443ede1daacb84c5"
server_key_iv: "3c56a3a31960da9fecd5a84af7a5e3fa"
server_key_version: "encryptionkey@0"
I20260927 03:32:00.362001 20719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:32:00.383950 20719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:32:00.385823 20719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:32:00.385980 20719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:32:00.390076 20719 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45731
I20260927 03:32:00.390128 20786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45731 every 8 connection(s)
I20260927 03:32:00.390543 20719 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:32:00.391382 20787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:32:00.393656 20787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a9260143cd74ac3bd4228547dbc4ecb: Bootstrap starting.
I20260927 03:32:00.394243 20787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a9260143cd74ac3bd4228547dbc4ecb: Neither blocks nor log segments found. Creating new log.
I20260927 03:32:00.394558 20787 log.cc:824] T 00000000000000000000000000000000 P 0a9260143cd74ac3bd4228547dbc4ecb: Log is configured to *not* fsync() on all Append() calls
I20260927 03:32:00.395222 20787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a9260143cd74ac3bd4228547dbc4ecb: No bootstrap required, opened a new log
I20260927 03:32:00.396786 20787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a9260143cd74ac3bd4228547dbc4ecb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a9260143cd74ac3bd4228547dbc4ecb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45731 } }
I20260927 03:32:00.396922 20787 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a9260143cd74ac3bd4228547dbc4ecb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:32:00.396966 20787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a9260143cd74ac3bd4228547dbc4ecb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a9260143cd74ac3bd4228547dbc4ecb, State: Initialized, Role: FOLLOWER
I20260927 03:32:00.397145 20787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a9260143cd74ac3bd4228547dbc4ecb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a9260143cd74ac3bd4228547dbc4ecb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45731 } }
I20260927 03:32:00.397210 20787 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a9260143cd74ac3bd4228547dbc4ecb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:32:00.397245 20787 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a9260143cd74ac3bd4228547dbc4ecb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:32:00.397294 20787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a9260143cd74ac3bd4228547dbc4ecb [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:32:00.397800 20787 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a9260143cd74ac3bd4228547dbc4ecb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a9260143cd74ac3bd4228547dbc4ecb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45731 } }
I20260927 03:32:00.397907 20787 leader_election.cc:302] T 00000000000000000000000000000000 P 0a9260143cd74ac3bd4228547dbc4ecb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a9260143cd74ac3bd4228547dbc4ecb; no voters:
I20260927 03:32:00.398088 20787 leader_election.cc:288] T 00000000000000000000000000000000 P 0a9260143cd74ac3bd4228547dbc4ecb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:32:00.398149 20790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a9260143cd74ac3bd4228547dbc4ecb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:32:00.398316 20790 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a9260143cd74ac3bd4228547dbc4ecb [term 1 LEADER]: Becoming Leader. State: Replica: 0a9260143cd74ac3bd4228547dbc4ecb, State: Running, Role: LEADER
I20260927 03:32:00.398341 20787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a9260143cd74ac3bd4228547dbc4ecb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:32:00.398424 20790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a9260143cd74ac3bd4228547dbc4ecb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a9260143cd74ac3bd4228547dbc4ecb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45731 } }
I20260927 03:32:00.398806 20791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a9260143cd74ac3bd4228547dbc4ecb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a9260143cd74ac3bd4228547dbc4ecb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a9260143cd74ac3bd4228547dbc4ecb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45731 } } }
I20260927 03:32:00.398819 20792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a9260143cd74ac3bd4228547dbc4ecb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a9260143cd74ac3bd4228547dbc4ecb. Latest consensus state: current_term: 1 leader_uuid: "0a9260143cd74ac3bd4228547dbc4ecb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a9260143cd74ac3bd4228547dbc4ecb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45731 } } }
I20260927 03:32:00.398954 20792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a9260143cd74ac3bd4228547dbc4ecb [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:00.399297 20791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a9260143cd74ac3bd4228547dbc4ecb [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:00.399415 20794 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:32:00.399979 20794 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:32:00.400723 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 20719
I20260927 03:32:00.400841 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:32:00.401062 392 external_mini_cluster.cc:1428] Setting key d3b521e6ac3324806e14f43780e1aeef
I20260927 03:32:00.401723 20794 catalog_manager.cc:1347] Generated new cluster ID: ebe97218546e40018f117cf72a5d1c2f
I20260927 03:32:00.401777 20794 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:32:00.407811 20810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:00.401498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47337 (local address 127.0.0.1:45731)
0927 03:32:00.403620 (+ 2122us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:00.403627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:00.403650 (+ 23us) server_negotiation.cc:407] Connection header received
0927 03:32:00.403745 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:00.403750 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:00.403798 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:00.403936 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:00.404147 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:00.405087 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:00.405848 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:00.406265 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:00.406379 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:00.407094 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:00.407097 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:00.407098 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:32:00.407270 (+ 172us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:00.407300 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:32:00.407311 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:00.407401 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:00.407581 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 03:32:00.407612 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2087,"thread_start_us":73,"threads_started":1}
I20260927 03:32:00.408761 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:32:00.414228 20810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:00.410089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60601 (local address 127.0.0.1:45731)
0927 03:32:00.410125 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:00.410127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:00.411593 (+ 1466us) server_negotiation.cc:407] Connection header received
0927 03:32:00.411656 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:00.411657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:00.411666 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:00.411721 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:00.411847 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:00.412317 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:00.413012 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:00.413239 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:00.413308 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:00.413975 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:00.413977 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:00.413977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:32:00.413981 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:00.413995 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:32:00.414000 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:00.414054 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:00.414119 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 03:32:00.414131 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:32:00.427611 20794 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:32:00.428334 20794 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:32:00.441404 20794 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a9260143cd74ac3bd4228547dbc4ecb: Generated new TSK 0
I20260927 03:32:00.441640 20794 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:32:00.449856 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 20719
W20260927 03:32:00.453881 20704 connection.cc:535] client connection to 127.0.0.1:45731 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45731): Connection reset by peer (error 104)
2026-09-27T03:32:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1341 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:32:00 dist-test-slave-j0hf krb5kdc[20818](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:32:00 dist-test-slave-j0hf krb5kdc[20818](info): set up 2 sockets
Sep 27 03:32:00 dist-test-slave-j0hf krb5kdc[20818](info): commencing operation
krb5kdc: starting...
W20260927 03:32:01.483515 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:32:01 dist-test-slave-j0hf krb5kdc[20818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:32:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:32:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:32:01.618466 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40285
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:32:01.700924 20831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:32:01.701138 20831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:32:01.701164 20831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:32:01.702359 20831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:32:01.702409 20831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:32:01.702425 20831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:32:01.702495 20831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:32:01.702523 20831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:32:01.704078 20831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40285
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:32:01.704337 20831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:32:01.704571 20831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:32:01.709448 20839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:32:01.709558 20831 server_base.cc:1034] running on GCE node
W20260927 03:32:01.709473 20837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:01.709653 20836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:32:01.709884 20831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:32:01.710156 20831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:32:01.711354 20831 hybrid_clock.cc:648] HybridClock initialized: now 1790479921711328 us; error 39 us; skew 500 ppm
Sep 27 03:32:01 dist-test-slave-j0hf krb5kdc[20818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:32:01.713793 20831 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:32:01.714234 20831 webserver.cc:460] Webserver started at http://127.0.0.1:37563/ using document root <none> and password file <none>
I20260927 03:32:01.714455 20831 fs_manager.cc:359] Metadata directory not provided
I20260927 03:32:01.714520 20831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:32:01.714671 20831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:32:01.715795 20831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "ca8e1258011449d398197a2108e7d763"
format_stamp: "Formatted at 2026-09-27 03:32:01 on dist-test-slave-j0hf"
server_key: "2b6d31ddddcafe38bb758371ba48f558"
server_key_iv: "cef1814c975b3cc02d4f3e86583d075f"
server_key_version: "encryptionkey@0"
I20260927 03:32:01.716148 20831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "ca8e1258011449d398197a2108e7d763"
format_stamp: "Formatted at 2026-09-27 03:32:01 on dist-test-slave-j0hf"
server_key: "2b6d31ddddcafe38bb758371ba48f558"
server_key_iv: "cef1814c975b3cc02d4f3e86583d075f"
server_key_version: "encryptionkey@0"
I20260927 03:32:01.717545 20831 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:32:01.718318 20846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:32:01.718528 20831 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:32:01.718622 20831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "ca8e1258011449d398197a2108e7d763"
format_stamp: "Formatted at 2026-09-27 03:32:01 on dist-test-slave-j0hf"
server_key: "2b6d31ddddcafe38bb758371ba48f558"
server_key_iv: "cef1814c975b3cc02d4f3e86583d075f"
server_key_version: "encryptionkey@0"
I20260927 03:32:01.718704 20831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:32:01.722118 20831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:32:01.724046 20831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:32:01.724195 20831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:32:01.727754 20831 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40285
I20260927 03:32:01.727783 20898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40285 every 8 connection(s)
I20260927 03:32:01.728076 20831 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:32:01.729035 20899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:32:01.731202 20899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca8e1258011449d398197a2108e7d763: Bootstrap starting.
I20260927 03:32:01.731801 20899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca8e1258011449d398197a2108e7d763: Neither blocks nor log segments found. Creating new log.
I20260927 03:32:01.732077 20899 log.cc:824] T 00000000000000000000000000000000 P ca8e1258011449d398197a2108e7d763: Log is configured to *not* fsync() on all Append() calls
I20260927 03:32:01.732692 20899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca8e1258011449d398197a2108e7d763: No bootstrap required, opened a new log
I20260927 03:32:01.733150 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 20831
I20260927 03:32:01.733278 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:32:01.733471 392 external_mini_cluster.cc:1428] Setting key 01471bf7f7e0d412915fa95b9062df72
I20260927 03:32:01.734869 20899 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca8e1258011449d398197a2108e7d763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca8e1258011449d398197a2108e7d763" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40285 } }
I20260927 03:32:01.735039 20899 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca8e1258011449d398197a2108e7d763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:32:01.735080 20899 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca8e1258011449d398197a2108e7d763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca8e1258011449d398197a2108e7d763, State: Initialized, Role: FOLLOWER
I20260927 03:32:01.735211 20899 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca8e1258011449d398197a2108e7d763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca8e1258011449d398197a2108e7d763" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40285 } }
I20260927 03:32:01.735303 20899 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca8e1258011449d398197a2108e7d763 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:32:01.735327 20899 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca8e1258011449d398197a2108e7d763 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:32:01.735354 20899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca8e1258011449d398197a2108e7d763 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:32:01.736435 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:01.733949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38641 (local address 127.0.0.1:40285)
0927 03:32:01.734140 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:01.734147 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:01.734171 (+ 24us) server_negotiation.cc:407] Connection header received
0927 03:32:01.734849 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:01.734857 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:01.734889 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:01.735002 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:01.735988 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:01.735992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:01.735993 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:32:01.735998 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:01.736052 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:32:01.736069 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:01.736153 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:01.736190 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 03:32:01.736229 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":100,"threads_started":1}
I20260927 03:32:01.736534 20899 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca8e1258011449d398197a2108e7d763 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca8e1258011449d398197a2108e7d763" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40285 } }
I20260927 03:32:01.736653 20899 leader_election.cc:302] T 00000000000000000000000000000000 P ca8e1258011449d398197a2108e7d763 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca8e1258011449d398197a2108e7d763; no voters:
I20260927 03:32:01.736825 20899 leader_election.cc:288] T 00000000000000000000000000000000 P ca8e1258011449d398197a2108e7d763 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:32:01.736905 20904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca8e1258011449d398197a2108e7d763 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:32:01.737044 20904 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca8e1258011449d398197a2108e7d763 [term 1 LEADER]: Becoming Leader. State: Replica: ca8e1258011449d398197a2108e7d763, State: Running, Role: LEADER
I20260927 03:32:01.737067 20899 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca8e1258011449d398197a2108e7d763 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:32:01.737135 20904 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca8e1258011449d398197a2108e7d763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca8e1258011449d398197a2108e7d763" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40285 } }
I20260927 03:32:01.737390 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:32:01.737468 20904 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca8e1258011449d398197a2108e7d763 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca8e1258011449d398197a2108e7d763. Latest consensus state: current_term: 1 leader_uuid: "ca8e1258011449d398197a2108e7d763" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca8e1258011449d398197a2108e7d763" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40285 } } }
I20260927 03:32:01.737561 20904 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca8e1258011449d398197a2108e7d763 [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:01.737480 20905 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca8e1258011449d398197a2108e7d763 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca8e1258011449d398197a2108e7d763" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca8e1258011449d398197a2108e7d763" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40285 } } }
I20260927 03:32:01.737751 20905 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca8e1258011449d398197a2108e7d763 [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:01.737874 20910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:32:01.738345 20910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:32:01.739791 20910 catalog_manager.cc:1347] Generated new cluster ID: 7484adb2802045a7aca5b85f24df2669
I20260927 03:32:01.739843 20910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:32:01.740145 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:01.739166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53607 (local address 127.0.0.1:40285)
0927 03:32:01.739202 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:01.739204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:01.739214 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:32:01.739234 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:01.739235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:01.739239 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:01.739293 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:01.739918 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:01.739919 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:01.739919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:32:01.739923 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:01.739937 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:32:01.739941 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:01.739975 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:01.740068 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 03:32:01.740083 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:32:01.752179 20910 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:32:01.752367 20910 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:32:01.755745 20910 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca8e1258011449d398197a2108e7d763: Generated new TSK 0
I20260927 03:32:01.755967 20910 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:32:01.765254 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 20831
2026-09-27T03:32:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:32:01 dist-test-slave-j0hf krb5kdc[20930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:32:01 dist-test-slave-j0hf krb5kdc[20930](info): set up 2 sockets
Sep 27 03:32:01 dist-test-slave-j0hf krb5kdc[20930](info): commencing operation
krb5kdc: starting...
W20260927 03:32:02.795621 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:32:02 dist-test-slave-j0hf krb5kdc[20930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:32:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:32:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:32:02.910545 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36507
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:32:02.989773 20943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:32:02.990010 20943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:32:02.990061 20943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:32:02.991191 20943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:32:02.991259 20943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:32:02.991281 20943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:32:02.991302 20943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:32:02.991330 20943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:32:02.992828 20943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:32:02.993103 20943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:32:02.993338 20943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:32:02.998230 20943 server_base.cc:1034] running on GCE node
W20260927 03:32:02.998205 20948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:02.998191 20951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:02.998199 20949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:32:02.998708 20943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:32:02.998942 20943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:32:03.000116 20943 hybrid_clock.cc:648] HybridClock initialized: now 1790479923000105 us; error 39 us; skew 500 ppm
Sep 27 03:32:02 dist-test-slave-j0hf krb5kdc[20930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:32:03.002693 20943 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:32:03.003147 20943 webserver.cc:460] Webserver started at http://127.0.0.1:40685/ using document root <none> and password file <none>
I20260927 03:32:03.003381 20943 fs_manager.cc:359] Metadata directory not provided
I20260927 03:32:03.003463 20943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:32:03.003634 20943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:32:03.004660 20943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "0acbf601b5714dd588613b9772fcbe21"
format_stamp: "Formatted at 2026-09-27 03:32:03 on dist-test-slave-j0hf"
server_key: "bb22c8122c766f850a0fe83141be5c85"
server_key_iv: "4dfe1e3c1927d66742f9f73488a53156"
server_key_version: "encryptionkey@0"
I20260927 03:32:03.004998 20943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "0acbf601b5714dd588613b9772fcbe21"
format_stamp: "Formatted at 2026-09-27 03:32:03 on dist-test-slave-j0hf"
server_key: "bb22c8122c766f850a0fe83141be5c85"
server_key_iv: "4dfe1e3c1927d66742f9f73488a53156"
server_key_version: "encryptionkey@0"
I20260927 03:32:03.006275 20943 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:32:03.007002 20958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:32:03.007185 20943 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:32:03.007279 20943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "0acbf601b5714dd588613b9772fcbe21"
format_stamp: "Formatted at 2026-09-27 03:32:03 on dist-test-slave-j0hf"
server_key: "bb22c8122c766f850a0fe83141be5c85"
server_key_iv: "4dfe1e3c1927d66742f9f73488a53156"
server_key_version: "encryptionkey@0"
I20260927 03:32:03.007360 20943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:32:03.016222 20943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:32:03.018433 20943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:32:03.018599 20943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:32:03.022535 20943 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36507
I20260927 03:32:03.022560 21010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36507 every 8 connection(s)
I20260927 03:32:03.022907 20943 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:32:03.023767 21011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:32:03.024693 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 20943
I20260927 03:32:03.024816 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:32:03.024984 392 external_mini_cluster.cc:1428] Setting key 9108e238065c45af2025c21b6b9476af
I20260927 03:32:03.027038 21011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0acbf601b5714dd588613b9772fcbe21: Bootstrap starting.
I20260927 03:32:03.027777 21011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0acbf601b5714dd588613b9772fcbe21: Neither blocks nor log segments found. Creating new log.
I20260927 03:32:03.028102 21011 log.cc:824] T 00000000000000000000000000000000 P 0acbf601b5714dd588613b9772fcbe21: Log is configured to *not* fsync() on all Append() calls
I20260927 03:32:03.028786 21011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0acbf601b5714dd588613b9772fcbe21: No bootstrap required, opened a new log
I20260927 03:32:03.030961 21011 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0acbf601b5714dd588613b9772fcbe21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0acbf601b5714dd588613b9772fcbe21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36507 } }
I20260927 03:32:03.031093 21011 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0acbf601b5714dd588613b9772fcbe21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:32:03.031116 21011 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0acbf601b5714dd588613b9772fcbe21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0acbf601b5714dd588613b9772fcbe21, State: Initialized, Role: FOLLOWER
I20260927 03:32:03.031260 21011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0acbf601b5714dd588613b9772fcbe21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0acbf601b5714dd588613b9772fcbe21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36507 } }
I20260927 03:32:03.031322 21011 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0acbf601b5714dd588613b9772fcbe21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:32:03.031343 21011 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0acbf601b5714dd588613b9772fcbe21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:32:03.031406 21011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0acbf601b5714dd588613b9772fcbe21 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 03:32:03 dist-test-slave-j0hf krb5kdc[20930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 03:32:03.031997 21011 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0acbf601b5714dd588613b9772fcbe21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0acbf601b5714dd588613b9772fcbe21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36507 } }
I20260927 03:32:03.032111 21011 leader_election.cc:302] T 00000000000000000000000000000000 P 0acbf601b5714dd588613b9772fcbe21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0acbf601b5714dd588613b9772fcbe21; no voters:
I20260927 03:32:03.032305 21011 leader_election.cc:288] T 00000000000000000000000000000000 P 0acbf601b5714dd588613b9772fcbe21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:32:03.032342 21016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0acbf601b5714dd588613b9772fcbe21 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:32:03.032466 21016 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0acbf601b5714dd588613b9772fcbe21 [term 1 LEADER]: Becoming Leader. State: Replica: 0acbf601b5714dd588613b9772fcbe21, State: Running, Role: LEADER
I20260927 03:32:03.032553 21016 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0acbf601b5714dd588613b9772fcbe21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0acbf601b5714dd588613b9772fcbe21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36507 } }
I20260927 03:32:03.032579 21011 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0acbf601b5714dd588613b9772fcbe21 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:32:03.033030 21016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0acbf601b5714dd588613b9772fcbe21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0acbf601b5714dd588613b9772fcbe21. Latest consensus state: current_term: 1 leader_uuid: "0acbf601b5714dd588613b9772fcbe21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0acbf601b5714dd588613b9772fcbe21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36507 } } }
I20260927 03:32:03.033142 21016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0acbf601b5714dd588613b9772fcbe21 [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:03.033051 21018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0acbf601b5714dd588613b9772fcbe21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0acbf601b5714dd588613b9772fcbe21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0acbf601b5714dd588613b9772fcbe21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36507 } } }
I20260927 03:32:03.033293 21018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0acbf601b5714dd588613b9772fcbe21 [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:03.033423 21023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:32:03.033962 21023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:32:03.035660 21023 catalog_manager.cc:1347] Generated new cluster ID: 1bd8a96af5504be293c1f571fb9355a3
I20260927 03:32:03.035751 21023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:32:03.037024 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:03.025509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43219 (local address 127.0.0.1:36507)
0927 03:32:03.025693 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:03.025703 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:03.025735 (+ 32us) server_negotiation.cc:407] Connection header received
0927 03:32:03.026257 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:03.026265 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:03.026332 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:03.026508 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:03.027090 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:03.028454 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:03.029295 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:03.029924 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:03.030050 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:03.032492 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:03.032496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:03.032496 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:03.032509 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:03.035534 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:03.035883 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:03.035886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:03.035886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:03.035982 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:03.036195 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:03.036196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:03.036196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:03.036431 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:03.036590 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:03.036708 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 03:32:03.036848 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":80,"threads_started":1}
I20260927 03:32:03.037961 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:32:03.045487 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:03.039287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48813 (local address 127.0.0.1:36507)
0927 03:32:03.039340 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:03.039341 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:03.039482 (+ 141us) server_negotiation.cc:407] Connection header received
0927 03:32:03.039560 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:03.039561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:03.039571 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:03.039627 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:03.040131 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:03.040618 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:03.041345 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:03.041601 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:03.041677 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:03.042872 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:03.042874 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:03.042874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:03.042881 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:03.044351 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:03.044713 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:03.044714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:03.044715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:03.044772 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:03.044966 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:03.044967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:03.044968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:03.045086 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:03.045183 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:03.045298 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 03:32:03.045376 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 03:32:03.059151 21023 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:32:03.059952 21023 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:32:03.066881 21023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0acbf601b5714dd588613b9772fcbe21: Generated new TSK 0
I20260927 03:32:03.067081 21023 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:32:03.069796 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 20943
2026-09-27T03:32:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:32:03 dist-test-slave-j0hf krb5kdc[21042](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:32:03 dist-test-slave-j0hf krb5kdc[21042](info): set up 2 sockets
Sep 27 03:32:03 dist-test-slave-j0hf krb5kdc[21042](info): commencing operation
krb5kdc: starting...
W20260927 03:32:04.095867 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:32:04 dist-test-slave-j0hf krb5kdc[21042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:32:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:32:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:32:04.210327 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41801
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:32:04.289522 21055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:32:04.289760 21055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:32:04.289798 21055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:32:04.290951 21055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:32:04.291018 21055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:32:04.291054 21055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:32:04.291096 21055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:32:04.291127 21055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:32:04.292661 21055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:32:04.292935 21055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:32:04.293166 21055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:32:04.298015 21063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:04.298032 21060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:04.298022 21061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:32:04.298038 21055 server_base.cc:1034] running on GCE node
I20260927 03:32:04.298462 21055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:32:04.298732 21055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:32:04.299901 21055 hybrid_clock.cc:648] HybridClock initialized: now 1790479924299885 us; error 53 us; skew 500 ppm
Sep 27 03:32:04 dist-test-slave-j0hf krb5kdc[21042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:32:04.302305 21055 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:32:04.302769 21055 webserver.cc:460] Webserver started at http://127.0.0.1:34597/ using document root <none> and password file <none>
I20260927 03:32:04.302997 21055 fs_manager.cc:359] Metadata directory not provided
I20260927 03:32:04.303076 21055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:32:04.303218 21055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:32:04.304294 21055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "e27cad41a78f49dda52aeba6ee599477"
format_stamp: "Formatted at 2026-09-27 03:32:04 on dist-test-slave-j0hf"
server_key: "f6612ca8c39fd318fcbbc03db9c09c16"
server_key_iv: "41c88da4bca382d1a5e4c340d6b14da9"
server_key_version: "encryptionkey@0"
I20260927 03:32:04.304644 21055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "e27cad41a78f49dda52aeba6ee599477"
format_stamp: "Formatted at 2026-09-27 03:32:04 on dist-test-slave-j0hf"
server_key: "f6612ca8c39fd318fcbbc03db9c09c16"
server_key_iv: "41c88da4bca382d1a5e4c340d6b14da9"
server_key_version: "encryptionkey@0"
I20260927 03:32:04.305907 21055 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:32:04.306586 21070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:32:04.306751 21055 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:32:04.306844 21055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "e27cad41a78f49dda52aeba6ee599477"
format_stamp: "Formatted at 2026-09-27 03:32:04 on dist-test-slave-j0hf"
server_key: "f6612ca8c39fd318fcbbc03db9c09c16"
server_key_iv: "41c88da4bca382d1a5e4c340d6b14da9"
server_key_version: "encryptionkey@0"
I20260927 03:32:04.306926 21055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:32:04.322299 21055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:32:04.324318 21055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:32:04.324484 21055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:32:04.329264 21055 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41801
I20260927 03:32:04.329298 21122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41801 every 8 connection(s)
I20260927 03:32:04.329662 21055 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:32:04.330534 21123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:32:04.332937 21123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e27cad41a78f49dda52aeba6ee599477: Bootstrap starting.
I20260927 03:32:04.333644 21123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e27cad41a78f49dda52aeba6ee599477: Neither blocks nor log segments found. Creating new log.
I20260927 03:32:04.333951 21123 log.cc:824] T 00000000000000000000000000000000 P e27cad41a78f49dda52aeba6ee599477: Log is configured to *not* fsync() on all Append() calls
I20260927 03:32:04.334627 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 21055
I20260927 03:32:04.334761 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:32:04.334928 392 external_mini_cluster.cc:1428] Setting key dc4b0682e9b5f932d691ea1793eab63c
I20260927 03:32:04.334926 21123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e27cad41a78f49dda52aeba6ee599477: No bootstrap required, opened a new log
I20260927 03:32:04.336721 21123 raft_consensus.cc:348] T 00000000000000000000000000000000 P e27cad41a78f49dda52aeba6ee599477 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e27cad41a78f49dda52aeba6ee599477" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41801 } }
I20260927 03:32:04.336886 21123 raft_consensus.cc:374] T 00000000000000000000000000000000 P e27cad41a78f49dda52aeba6ee599477 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:32:04.336934 21123 raft_consensus.cc:729] T 00000000000000000000000000000000 P e27cad41a78f49dda52aeba6ee599477 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e27cad41a78f49dda52aeba6ee599477, State: Initialized, Role: FOLLOWER
I20260927 03:32:04.337072 21123 consensus_queue.cc:260] T 00000000000000000000000000000000 P e27cad41a78f49dda52aeba6ee599477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e27cad41a78f49dda52aeba6ee599477" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41801 } }
I20260927 03:32:04.337159 21123 raft_consensus.cc:388] T 00000000000000000000000000000000 P e27cad41a78f49dda52aeba6ee599477 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:32:04.337198 21123 raft_consensus.cc:482] T 00000000000000000000000000000000 P e27cad41a78f49dda52aeba6ee599477 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:32:04.337242 21123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e27cad41a78f49dda52aeba6ee599477 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:32:04.338433 21123 raft_consensus.cc:504] T 00000000000000000000000000000000 P e27cad41a78f49dda52aeba6ee599477 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e27cad41a78f49dda52aeba6ee599477" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41801 } }
I20260927 03:32:04.338550 21123 leader_election.cc:302] T 00000000000000000000000000000000 P e27cad41a78f49dda52aeba6ee599477 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e27cad41a78f49dda52aeba6ee599477; no voters:
I20260927 03:32:04.338708 21123 leader_election.cc:288] T 00000000000000000000000000000000 P e27cad41a78f49dda52aeba6ee599477 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:32:04.338755 21128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e27cad41a78f49dda52aeba6ee599477 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:32:04.338922 21128 raft_consensus.cc:686] T 00000000000000000000000000000000 P e27cad41a78f49dda52aeba6ee599477 [term 1 LEADER]: Becoming Leader. State: Replica: e27cad41a78f49dda52aeba6ee599477, State: Running, Role: LEADER
I20260927 03:32:04.338968 21123 sys_catalog.cc:564] T 00000000000000000000000000000000 P e27cad41a78f49dda52aeba6ee599477 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:32:04.339003 21128 consensus_queue.cc:237] T 00000000000000000000000000000000 P e27cad41a78f49dda52aeba6ee599477 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e27cad41a78f49dda52aeba6ee599477" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41801 } }
I20260927 03:32:04.339323 21128 sys_catalog.cc:455] T 00000000000000000000000000000000 P e27cad41a78f49dda52aeba6ee599477 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e27cad41a78f49dda52aeba6ee599477. Latest consensus state: current_term: 1 leader_uuid: "e27cad41a78f49dda52aeba6ee599477" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e27cad41a78f49dda52aeba6ee599477" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41801 } } }
I20260927 03:32:04.339358 21129 sys_catalog.cc:455] T 00000000000000000000000000000000 P e27cad41a78f49dda52aeba6ee599477 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e27cad41a78f49dda52aeba6ee599477" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e27cad41a78f49dda52aeba6ee599477" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41801 } } }
I20260927 03:32:04.339394 21128 sys_catalog.cc:458] T 00000000000000000000000000000000 P e27cad41a78f49dda52aeba6ee599477 [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:04.339409 21129 sys_catalog.cc:458] T 00000000000000000000000000000000 P e27cad41a78f49dda52aeba6ee599477 [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:04.339697 21134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:32:04.340227 21134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:32:04.342507 21134 catalog_manager.cc:1347] Generated new cluster ID: bdf022375c894b6fbd0f84ee64b36ee4
I20260927 03:32:04.342581 21134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:32:04.345712 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:04.335481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51251 (local address 127.0.0.1:41801)
0927 03:32:04.340523 (+ 5042us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:04.340535 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:04.340582 (+ 47us) server_negotiation.cc:407] Connection header received
0927 03:32:04.340709 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:04.340717 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:04.340776 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:04.340921 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:04.341192 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:04.342138 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:04.343520 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:04.344138 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:04.344241 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:04.345033 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:04.345037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:04.345038 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:32:04.345232 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:04.345267 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:32:04.345281 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:04.345341 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:04.345450 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 03:32:04.345508 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4984,"thread_start_us":90,"threads_started":1}
I20260927 03:32:04.346814 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:32:04.352383 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:04.348983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40601 (local address 127.0.0.1:41801)
0927 03:32:04.349034 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:04.349037 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:04.349046 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:32:04.349064 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:04.349065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:04.349098 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:04.349145 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:04.349372 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:04.350088 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:04.351045 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:04.351329 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:04.351363 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:04.352144 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:04.352145 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:04.352145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:32:04.352151 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:04.352164 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:32:04.352168 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:04.352199 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:04.352290 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 03:32:04.352313 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 03:32:04.367718 21134 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:32:04.368537 21134 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:32:04.375236 21134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e27cad41a78f49dda52aeba6ee599477: Generated new TSK 0
I20260927 03:32:04.375465 21134 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:32:04.381145 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 21055
2026-09-27T03:32:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:32:04 dist-test-slave-j0hf krb5kdc[21154](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:32:04 dist-test-slave-j0hf krb5kdc[21154](info): set up 2 sockets
Sep 27 03:32:04 dist-test-slave-j0hf krb5kdc[21154](info): commencing operation
krb5kdc: starting...
W20260927 03:32:05.415668 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:32:05 dist-test-slave-j0hf krb5kdc[21154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:32:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:32:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:32:05.526261 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37137
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:32:05.606192 21167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:32:05.606410 21167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:32:05.606446 21167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:32:05.607681 21167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:32:05.607738 21167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:32:05.607765 21167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:32:05.607807 21167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:32:05.607838 21167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:32:05.609285 21167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:32:05.609560 21167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:32:05.609773 21167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:32:05.614526 21172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:05.614557 21173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:05.614552 21175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:32:05.614773 21167 server_base.cc:1034] running on GCE node
I20260927 03:32:05.615010 21167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:32:05.615325 21167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:32:05.616477 21167 hybrid_clock.cc:648] HybridClock initialized: now 1790479925616478 us; error 48 us; skew 500 ppm
Sep 27 03:32:05 dist-test-slave-j0hf krb5kdc[21154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:32:05.632634 21167 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:32:05.633078 21167 webserver.cc:460] Webserver started at http://127.0.0.1:46059/ using document root <none> and password file <none>
I20260927 03:32:05.633296 21167 fs_manager.cc:359] Metadata directory not provided
I20260927 03:32:05.633356 21167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:32:05.633497 21167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:32:05.634558 21167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "cb329545e29947ca9e5392f4f0c11c76"
format_stamp: "Formatted at 2026-09-27 03:32:05 on dist-test-slave-j0hf"
server_key: "a99c184b72010d5ed54a2b9500c3bab9"
server_key_iv: "c3181d2b872311f36c40831904f73985"
server_key_version: "encryptionkey@0"
I20260927 03:32:05.634923 21167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "cb329545e29947ca9e5392f4f0c11c76"
format_stamp: "Formatted at 2026-09-27 03:32:05 on dist-test-slave-j0hf"
server_key: "a99c184b72010d5ed54a2b9500c3bab9"
server_key_iv: "c3181d2b872311f36c40831904f73985"
server_key_version: "encryptionkey@0"
I20260927 03:32:05.636318 21167 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:32:05.637018 21182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:32:05.637184 21167 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:32:05.637285 21167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "cb329545e29947ca9e5392f4f0c11c76"
format_stamp: "Formatted at 2026-09-27 03:32:05 on dist-test-slave-j0hf"
server_key: "a99c184b72010d5ed54a2b9500c3bab9"
server_key_iv: "c3181d2b872311f36c40831904f73985"
server_key_version: "encryptionkey@0"
I20260927 03:32:05.637355 21167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:32:05.640702 21167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:32:05.643075 21167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:32:05.643224 21167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:32:05.647084 21167 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37137
I20260927 03:32:05.647111 21234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37137 every 8 connection(s)
I20260927 03:32:05.647493 21167 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:32:05.648439 21235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:32:05.650813 21235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb329545e29947ca9e5392f4f0c11c76: Bootstrap starting.
I20260927 03:32:05.650878 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 21167
I20260927 03:32:05.650997 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:32:05.651190 392 external_mini_cluster.cc:1428] Setting key 83b63261582b2774ff6001bf2ae99093
I20260927 03:32:05.651604 21235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb329545e29947ca9e5392f4f0c11c76: Neither blocks nor log segments found. Creating new log.
I20260927 03:32:05.651998 21235 log.cc:824] T 00000000000000000000000000000000 P cb329545e29947ca9e5392f4f0c11c76: Log is configured to *not* fsync() on all Append() calls
I20260927 03:32:05.652736 21235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb329545e29947ca9e5392f4f0c11c76: No bootstrap required, opened a new log
I20260927 03:32:05.653625 21238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:05.651737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49547 (local address 127.0.0.1:37137)
0927 03:32:05.651935 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:05.651944 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:05.651974 (+ 30us) server_negotiation.cc:407] Connection header received
0927 03:32:05.652455 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:05.652461 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:05.652495 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:05.652613 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:05.653210 (+ 597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:05.653213 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:05.653213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:32:05.653216 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:05.653247 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:32:05.653256 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:05.653318 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:05.653382 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 03:32:05.653434 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":93,"threads_started":1}
I20260927 03:32:05.654508 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:32:05.655099 21235 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb329545e29947ca9e5392f4f0c11c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb329545e29947ca9e5392f4f0c11c76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37137 } }
I20260927 03:32:05.655228 21235 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb329545e29947ca9e5392f4f0c11c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:32:05.655249 21235 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb329545e29947ca9e5392f4f0c11c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb329545e29947ca9e5392f4f0c11c76, State: Initialized, Role: FOLLOWER
I20260927 03:32:05.655359 21235 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb329545e29947ca9e5392f4f0c11c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb329545e29947ca9e5392f4f0c11c76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37137 } }
I20260927 03:32:05.655493 21235 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb329545e29947ca9e5392f4f0c11c76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:32:05.655539 21235 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb329545e29947ca9e5392f4f0c11c76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:32:05.655591 21235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb329545e29947ca9e5392f4f0c11c76 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:32:05.656325 21235 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb329545e29947ca9e5392f4f0c11c76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb329545e29947ca9e5392f4f0c11c76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37137 } }
I20260927 03:32:05.656435 21235 leader_election.cc:302] T 00000000000000000000000000000000 P cb329545e29947ca9e5392f4f0c11c76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb329545e29947ca9e5392f4f0c11c76; no voters:
I20260927 03:32:05.656612 21235 leader_election.cc:288] T 00000000000000000000000000000000 P cb329545e29947ca9e5392f4f0c11c76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:32:05.656649 21245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb329545e29947ca9e5392f4f0c11c76 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:32:05.656826 21245 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb329545e29947ca9e5392f4f0c11c76 [term 1 LEADER]: Becoming Leader. State: Replica: cb329545e29947ca9e5392f4f0c11c76, State: Running, Role: LEADER
I20260927 03:32:05.656878 21235 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb329545e29947ca9e5392f4f0c11c76 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:32:05.656980 21245 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb329545e29947ca9e5392f4f0c11c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb329545e29947ca9e5392f4f0c11c76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37137 } }
I20260927 03:32:05.657181 21238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:05.655900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50241 (local address 127.0.0.1:37137)
0927 03:32:05.655956 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:05.655958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:05.656095 (+ 137us) server_negotiation.cc:407] Connection header received
0927 03:32:05.656152 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:05.656153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:05.656157 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:05.656203 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:05.656798 (+ 595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:05.656799 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:05.656799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:32:05.656804 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:05.656818 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:32:05.656822 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:05.656902 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:05.657117 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 03:32:05.657130 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 03:32:05.657292 21245 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb329545e29947ca9e5392f4f0c11c76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb329545e29947ca9e5392f4f0c11c76. Latest consensus state: current_term: 1 leader_uuid: "cb329545e29947ca9e5392f4f0c11c76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb329545e29947ca9e5392f4f0c11c76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37137 } } }
I20260927 03:32:05.657302 21247 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb329545e29947ca9e5392f4f0c11c76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb329545e29947ca9e5392f4f0c11c76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb329545e29947ca9e5392f4f0c11c76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37137 } } }
I20260927 03:32:05.657395 21245 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb329545e29947ca9e5392f4f0c11c76 [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:05.657454 21247 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb329545e29947ca9e5392f4f0c11c76 [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:05.657665 21251 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:32:05.658193 21251 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:32:05.659703 21251 catalog_manager.cc:1347] Generated new cluster ID: 0c14381ae8cd48a2ba330686647a95f5
I20260927 03:32:05.659756 21251 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:32:05.680210 21251 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:32:05.680388 21251 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:32:05.692963 21251 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb329545e29947ca9e5392f4f0c11c76: Generated new TSK 0
I20260927 03:32:05.693161 21251 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:32:05.701813 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 21167
2026-09-27T03:32:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:32:05 dist-test-slave-j0hf krb5kdc[21266](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:32:05 dist-test-slave-j0hf krb5kdc[21266](info): set up 2 sockets
Sep 27 03:32:05 dist-test-slave-j0hf krb5kdc[21266](info): commencing operation
krb5kdc: starting...
W20260927 03:32:06.747409 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:32:06 dist-test-slave-j0hf krb5kdc[21266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:32:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:32:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:32:06.861917 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35715
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:32:06.941927 21279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:32:06.942144 21279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:32:06.942169 21279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:32:06.943264 21279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:32:06.943296 21279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:32:06.943310 21279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:32:06.943322 21279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:32:06.943333 21279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:32:06.944785 21279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:32:06.944993 21279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:32:06.945158 21279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:32:06.950083 21285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:32:06.950173 21279 server_base.cc:1034] running on GCE node
W20260927 03:32:06.950139 21287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:06.950101 21284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:32:06.950562 21279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:32:06.950806 21279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:32:06.951949 21279 hybrid_clock.cc:648] HybridClock initialized: now 1790479926951925 us; error 40 us; skew 500 ppm
Sep 27 03:32:06 dist-test-slave-j0hf krb5kdc[21266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:32:06.954372 21279 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:32:06.954862 21279 webserver.cc:460] Webserver started at http://127.0.0.1:36623/ using document root <none> and password file <none>
I20260927 03:32:06.955092 21279 fs_manager.cc:359] Metadata directory not provided
I20260927 03:32:06.955163 21279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:32:06.955314 21279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:32:06.956413 21279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "421fe7989b124c47bd571f7d88da7a57"
format_stamp: "Formatted at 2026-09-27 03:32:06 on dist-test-slave-j0hf"
server_key: "7879dcdcc94bbf1fda127e8db08832ae"
server_key_iv: "794f6fef0e521408239c496d1e49d81c"
server_key_version: "encryptionkey@0"
I20260927 03:32:06.956784 21279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "421fe7989b124c47bd571f7d88da7a57"
format_stamp: "Formatted at 2026-09-27 03:32:06 on dist-test-slave-j0hf"
server_key: "7879dcdcc94bbf1fda127e8db08832ae"
server_key_iv: "794f6fef0e521408239c496d1e49d81c"
server_key_version: "encryptionkey@0"
I20260927 03:32:06.958091 21279 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:32:06.958854 21294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:32:06.959053 21279 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:32:06.959161 21279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "421fe7989b124c47bd571f7d88da7a57"
format_stamp: "Formatted at 2026-09-27 03:32:06 on dist-test-slave-j0hf"
server_key: "7879dcdcc94bbf1fda127e8db08832ae"
server_key_iv: "794f6fef0e521408239c496d1e49d81c"
server_key_version: "encryptionkey@0"
I20260927 03:32:06.959245 21279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:32:06.976016 21279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:32:06.977842 21279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:32:06.978013 21279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:32:06.982030 21279 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35715
I20260927 03:32:06.982072 21346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35715 every 8 connection(s)
I20260927 03:32:06.982445 21279 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:32:06.983315 21347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:32:06.985571 21347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 421fe7989b124c47bd571f7d88da7a57: Bootstrap starting.
I20260927 03:32:06.986129 21347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 421fe7989b124c47bd571f7d88da7a57: Neither blocks nor log segments found. Creating new log.
I20260927 03:32:06.986378 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 21279
I20260927 03:32:06.986521 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:32:06.986507 21347 log.cc:824] T 00000000000000000000000000000000 P 421fe7989b124c47bd571f7d88da7a57: Log is configured to *not* fsync() on all Append() calls
I20260927 03:32:06.986733 392 external_mini_cluster.cc:1428] Setting key 5253f6f6e3619535f03854a79aa21884
I20260927 03:32:06.987375 21347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 421fe7989b124c47bd571f7d88da7a57: No bootstrap required, opened a new log
I20260927 03:32:06.989559 21347 raft_consensus.cc:348] T 00000000000000000000000000000000 P 421fe7989b124c47bd571f7d88da7a57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "421fe7989b124c47bd571f7d88da7a57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35715 } }
I20260927 03:32:06.989717 21347 raft_consensus.cc:374] T 00000000000000000000000000000000 P 421fe7989b124c47bd571f7d88da7a57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:32:06.989780 21347 raft_consensus.cc:729] T 00000000000000000000000000000000 P 421fe7989b124c47bd571f7d88da7a57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 421fe7989b124c47bd571f7d88da7a57, State: Initialized, Role: FOLLOWER
I20260927 03:32:06.989909 21347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 421fe7989b124c47bd571f7d88da7a57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "421fe7989b124c47bd571f7d88da7a57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35715 } }
I20260927 03:32:06.990033 21347 raft_consensus.cc:388] T 00000000000000000000000000000000 P 421fe7989b124c47bd571f7d88da7a57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:32:06.990077 21347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 421fe7989b124c47bd571f7d88da7a57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:32:06.990126 21347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 421fe7989b124c47bd571f7d88da7a57 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:32:06.990643 21347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 421fe7989b124c47bd571f7d88da7a57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "421fe7989b124c47bd571f7d88da7a57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35715 } }
I20260927 03:32:06.990772 21347 leader_election.cc:302] T 00000000000000000000000000000000 P 421fe7989b124c47bd571f7d88da7a57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 421fe7989b124c47bd571f7d88da7a57; no voters:
I20260927 03:32:06.990967 21347 leader_election.cc:288] T 00000000000000000000000000000000 P 421fe7989b124c47bd571f7d88da7a57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:32:06.991003 21352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 421fe7989b124c47bd571f7d88da7a57 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:32:06.991197 21352 raft_consensus.cc:686] T 00000000000000000000000000000000 P 421fe7989b124c47bd571f7d88da7a57 [term 1 LEADER]: Becoming Leader. State: Replica: 421fe7989b124c47bd571f7d88da7a57, State: Running, Role: LEADER
I20260927 03:32:06.991204 21347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 421fe7989b124c47bd571f7d88da7a57 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:32:06.991353 21352 consensus_queue.cc:237] T 00000000000000000000000000000000 P 421fe7989b124c47bd571f7d88da7a57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "421fe7989b124c47bd571f7d88da7a57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35715 } }
I20260927 03:32:06.992002 21354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 421fe7989b124c47bd571f7d88da7a57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 421fe7989b124c47bd571f7d88da7a57. Latest consensus state: current_term: 1 leader_uuid: "421fe7989b124c47bd571f7d88da7a57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "421fe7989b124c47bd571f7d88da7a57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35715 } } }
I20260927 03:32:06.992127 21354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 421fe7989b124c47bd571f7d88da7a57 [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:06.992375 21362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:32:06.992667 21353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 421fe7989b124c47bd571f7d88da7a57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "421fe7989b124c47bd571f7d88da7a57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "421fe7989b124c47bd571f7d88da7a57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35715 } } }
I20260927 03:32:06.992795 21353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 421fe7989b124c47bd571f7d88da7a57 [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:06.992897 21362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 03:32:06 dist-test-slave-j0hf krb5kdc[21266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 03:32:06.994575 21362 catalog_manager.cc:1347] Generated new cluster ID: 3c9a92e4a1a446d3847207b3f9d87ca2
I20260927 03:32:06.994643 21362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:32:06.997684 21351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:06.987268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:42361 (local address 127.0.0.1:35715)
0927 03:32:06.987476 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:06.987486 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:06.987521 (+ 35us) server_negotiation.cc:407] Connection header received
0927 03:32:06.987946 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:06.987951 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:06.987995 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:06.988136 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:06.988781 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:06.989653 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:06.990579 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:06.991181 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:06.991287 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:06.994389 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:06.994393 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:06.994393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:06.994404 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:06.996126 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:06.996517 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:06.996520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:06.996520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:06.996611 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:06.996849 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:06.996850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:06.996850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:06.997090 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:06.997273 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:06.997366 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 03:32:06.997470 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
I20260927 03:32:06.998894 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:32:07.001078 21362 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:32:07.001765 21362 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:32:07.009485 21362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 421fe7989b124c47bd571f7d88da7a57: Generated new TSK 0
I20260927 03:32:07.009693 21362 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:32:07.010171 21351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:07.000305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:56865 (local address 127.0.0.1:35715)
0927 03:32:07.000342 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:07.000344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:07.000749 (+ 405us) server_negotiation.cc:407] Connection header received
0927 03:32:07.000799 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:07.000799 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:07.000815 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:07.000875 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:07.004072 (+ 3197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:07.004561 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:07.005357 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:07.005555 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:07.005631 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:07.006785 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:07.006786 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:07.006787 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:07.006794 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:07.008336 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:07.008708 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:07.008709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:07.008709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:07.008781 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:07.009041 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:07.009042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:07.009043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:07.009161 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:07.009269 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:07.009977 (+ 708us) server_negotiation.cc:299] Negotiation successful
0927 03:32:07.010048 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:32:07.012176 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 21279
W20260927 03:32:07.016163 21264 connection.cc:535] client connection to 127.0.0.1:35715 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35715): Connection reset by peer (error 104)
2026-09-27T03:32:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:32:07 dist-test-slave-j0hf krb5kdc[21378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:32:07 dist-test-slave-j0hf krb5kdc[21378](info): set up 2 sockets
Sep 27 03:32:07 dist-test-slave-j0hf krb5kdc[21378](info): commencing operation
krb5kdc: starting...
W20260927 03:32:08.043708 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:32:08 dist-test-slave-j0hf krb5kdc[21378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:32:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:32:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:32:08.160599 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36817
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:32:08.240607 21391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:32:08.240792 21391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:32:08.240814 21391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:32:08.241909 21391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:32:08.241951 21391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:32:08.241964 21391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:32:08.241977 21391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:32:08.241989 21391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:32:08.243453 21391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:32:08.243716 21391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:32:08.243940 21391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:32:08.248627 21396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:08.248627 21399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:08.248651 21397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:32:08.248950 21391 server_base.cc:1034] running on GCE node
I20260927 03:32:08.249208 21391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:32:08.249513 21391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:32:08.250694 21391 hybrid_clock.cc:648] HybridClock initialized: now 1790479928250677 us; error 52 us; skew 500 ppm
Sep 27 03:32:08 dist-test-slave-j0hf krb5kdc[21378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:32:08.253214 21391 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:32:08.253588 21391 webserver.cc:460] Webserver started at http://127.0.0.1:33117/ using document root <none> and password file <none>
I20260927 03:32:08.253829 21391 fs_manager.cc:359] Metadata directory not provided
I20260927 03:32:08.253883 21391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:32:08.254050 21391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:32:08.255234 21391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "61eda8baf43b44d7a97e1665b951c243"
format_stamp: "Formatted at 2026-09-27 03:32:08 on dist-test-slave-j0hf"
server_key: "95508390bd828f3528727920c89312e7"
server_key_iv: "0f5f83803ba1d198587488fddf585fe1"
server_key_version: "encryptionkey@0"
I20260927 03:32:08.255622 21391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "61eda8baf43b44d7a97e1665b951c243"
format_stamp: "Formatted at 2026-09-27 03:32:08 on dist-test-slave-j0hf"
server_key: "95508390bd828f3528727920c89312e7"
server_key_iv: "0f5f83803ba1d198587488fddf585fe1"
server_key_version: "encryptionkey@0"
I20260927 03:32:08.256966 21391 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:32:08.257689 21406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:32:08.257889 21391 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:32:08.258000 21391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "61eda8baf43b44d7a97e1665b951c243"
format_stamp: "Formatted at 2026-09-27 03:32:08 on dist-test-slave-j0hf"
server_key: "95508390bd828f3528727920c89312e7"
server_key_iv: "0f5f83803ba1d198587488fddf585fe1"
server_key_version: "encryptionkey@0"
I20260927 03:32:08.258074 21391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:32:08.265286 21391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:32:08.267701 21391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:32:08.267869 21391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:32:08.271653 21391 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36817
I20260927 03:32:08.271716 21458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36817 every 8 connection(s)
I20260927 03:32:08.272071 21391 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:32:08.272939 21459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:32:08.275152 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 21391
I20260927 03:32:08.275278 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:32:08.275302 21459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61eda8baf43b44d7a97e1665b951c243: Bootstrap starting.
I20260927 03:32:08.275502 392 external_mini_cluster.cc:1428] Setting key bf7aa9ba97a8a51f0258530ae2b938cd
I20260927 03:32:08.276072 21459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61eda8baf43b44d7a97e1665b951c243: Neither blocks nor log segments found. Creating new log.
I20260927 03:32:08.276433 21459 log.cc:824] T 00000000000000000000000000000000 P 61eda8baf43b44d7a97e1665b951c243: Log is configured to *not* fsync() on all Append() calls
I20260927 03:32:08.277242 21459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61eda8baf43b44d7a97e1665b951c243: No bootstrap required, opened a new log
I20260927 03:32:08.279449 21459 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61eda8baf43b44d7a97e1665b951c243 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61eda8baf43b44d7a97e1665b951c243" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36817 } }
I20260927 03:32:08.279620 21459 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61eda8baf43b44d7a97e1665b951c243 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:32:08.279644 21459 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61eda8baf43b44d7a97e1665b951c243 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61eda8baf43b44d7a97e1665b951c243, State: Initialized, Role: FOLLOWER
I20260927 03:32:08.279811 21459 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61eda8baf43b44d7a97e1665b951c243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61eda8baf43b44d7a97e1665b951c243" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36817 } }
I20260927 03:32:08.279888 21459 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61eda8baf43b44d7a97e1665b951c243 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:32:08.279909 21459 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61eda8baf43b44d7a97e1665b951c243 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:32:08.279975 21459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61eda8baf43b44d7a97e1665b951c243 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:32:08.280787 21459 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61eda8baf43b44d7a97e1665b951c243 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61eda8baf43b44d7a97e1665b951c243" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36817 } }
I20260927 03:32:08.280907 21459 leader_election.cc:302] T 00000000000000000000000000000000 P 61eda8baf43b44d7a97e1665b951c243 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61eda8baf43b44d7a97e1665b951c243; no voters:
I20260927 03:32:08.281113 21459 leader_election.cc:288] T 00000000000000000000000000000000 P 61eda8baf43b44d7a97e1665b951c243 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:32:08.281150 21464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61eda8baf43b44d7a97e1665b951c243 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:32:08.281311 21464 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61eda8baf43b44d7a97e1665b951c243 [term 1 LEADER]: Becoming Leader. State: Replica: 61eda8baf43b44d7a97e1665b951c243, State: Running, Role: LEADER
I20260927 03:32:08.281378 21459 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61eda8baf43b44d7a97e1665b951c243 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:32:08.281492 21464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61eda8baf43b44d7a97e1665b951c243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61eda8baf43b44d7a97e1665b951c243" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36817 } }
I20260927 03:32:08.281920 21465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61eda8baf43b44d7a97e1665b951c243 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61eda8baf43b44d7a97e1665b951c243" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61eda8baf43b44d7a97e1665b951c243" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36817 } } }
I20260927 03:32:08.281968 21462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:08.276010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:33337 (local address 127.0.0.1:36817)
0927 03:32:08.276198 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:08.276206 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:08.276231 (+ 25us) server_negotiation.cc:407] Connection header received
0927 03:32:08.276793 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:08.276803 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:08.276932 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:08.277052 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:08.277337 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:08.278218 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:08.279509 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:08.280113 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:08.280208 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:08.281069 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:08.281073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:08.281073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:32:08.281296 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:08.281337 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:32:08.281354 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:08.281686 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:08.281731 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 03:32:08.281761 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
I20260927 03:32:08.282079 21466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61eda8baf43b44d7a97e1665b951c243 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61eda8baf43b44d7a97e1665b951c243. Latest consensus state: current_term: 1 leader_uuid: "61eda8baf43b44d7a97e1665b951c243" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61eda8baf43b44d7a97e1665b951c243" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36817 } } }
I20260927 03:32:08.282023 21465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61eda8baf43b44d7a97e1665b951c243 [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:08.282840 21466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61eda8baf43b44d7a97e1665b951c243 [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:08.283298 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260927 03:32:08.283532 21480 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 61eda8baf43b44d7a97e1665b951c243: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:32:08.283591 21480 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:32:08.283650 21474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:32:08.283816 21474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:32:08.285422 21474 catalog_manager.cc:1347] Generated new cluster ID: 21412dba6ab74324bb28ede7c2055be7
I20260927 03:32:08.285486 21474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:32:08.288167 21462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:08.284946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:59081 (local address 127.0.0.1:36817)
0927 03:32:08.284984 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:08.284986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:08.285223 (+ 237us) server_negotiation.cc:407] Connection header received
0927 03:32:08.285272 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:08.285273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:08.285285 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:08.285344 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:08.285510 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:08.286004 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:08.286979 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:08.287185 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:08.287242 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:08.287900 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:08.287901 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:08.287901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:32:08.287906 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:08.287919 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:32:08.287923 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:08.287975 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:08.288047 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 03:32:08.288056 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:32:08.299661 21474 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:32:08.300211 21474 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:32:08.302966 21474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61eda8baf43b44d7a97e1665b951c243: Generated new TSK 0
I20260927 03:32:08.303148 21474 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:32:08.307505 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 21391
W20260927 03:32:08.313694 21376 connection.cc:535] client connection to 127.0.0.1:36817 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36817): Connection reset by peer (error 104)
2026-09-27T03:32:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:32:08 dist-test-slave-j0hf krb5kdc[21490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:32:08 dist-test-slave-j0hf krb5kdc[21490](info): set up 2 sockets
Sep 27 03:32:08 dist-test-slave-j0hf krb5kdc[21490](info): commencing operation
krb5kdc: starting...
W20260927 03:32:09.339720 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:32:09 dist-test-slave-j0hf krb5kdc[21490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:32:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:32:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:32:09.458350 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36809
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:32:09.538157 21503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:32:09.538432 21503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:32:09.538475 21503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:32:09.539685 21503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:32:09.539757 21503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:32:09.539781 21503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:32:09.539811 21503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:32:09.539842 21503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:32:09.541324 21503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:32:09.541622 21503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:32:09.541852 21503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:32:09.546850 21511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:09.546849 21509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:32:09.547081 21503 server_base.cc:1034] running on GCE node
W20260927 03:32:09.547123 21508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:32:09.547372 21503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:32:09.547686 21503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:32:09.548837 21503 hybrid_clock.cc:648] HybridClock initialized: now 1790479929548813 us; error 33 us; skew 500 ppm
Sep 27 03:32:09 dist-test-slave-j0hf krb5kdc[21490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:32:09.551335 21503 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:32:09.551843 21503 webserver.cc:460] Webserver started at http://127.0.0.1:46177/ using document root <none> and password file <none>
I20260927 03:32:09.552039 21503 fs_manager.cc:359] Metadata directory not provided
I20260927 03:32:09.552079 21503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:32:09.552167 21503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:32:09.553151 21503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "15457badb08d4247801d70d30366e9c0"
format_stamp: "Formatted at 2026-09-27 03:32:09 on dist-test-slave-j0hf"
server_key: "e1aced7a6f46f2824456272896b96d5a"
server_key_iv: "c57a775bc98692c90477a85c5c2206c4"
server_key_version: "encryptionkey@0"
I20260927 03:32:09.553452 21503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "15457badb08d4247801d70d30366e9c0"
format_stamp: "Formatted at 2026-09-27 03:32:09 on dist-test-slave-j0hf"
server_key: "e1aced7a6f46f2824456272896b96d5a"
server_key_iv: "c57a775bc98692c90477a85c5c2206c4"
server_key_version: "encryptionkey@0"
I20260927 03:32:09.554666 21503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:32:09.555315 21518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:32:09.555500 21503 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:32:09.555604 21503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "15457badb08d4247801d70d30366e9c0"
format_stamp: "Formatted at 2026-09-27 03:32:09 on dist-test-slave-j0hf"
server_key: "e1aced7a6f46f2824456272896b96d5a"
server_key_iv: "c57a775bc98692c90477a85c5c2206c4"
server_key_version: "encryptionkey@0"
I20260927 03:32:09.555676 21503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:32:09.558903 21503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:32:09.561199 21503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:32:09.561354 21503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:32:09.565212 21503 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36809
I20260927 03:32:09.565240 21570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36809 every 8 connection(s)
I20260927 03:32:09.565555 21503 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:32:09.566466 21571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:32:09.568794 21571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15457badb08d4247801d70d30366e9c0: Bootstrap starting.
I20260927 03:32:09.569365 21571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15457badb08d4247801d70d30366e9c0: Neither blocks nor log segments found. Creating new log.
I20260927 03:32:09.569651 21571 log.cc:824] T 00000000000000000000000000000000 P 15457badb08d4247801d70d30366e9c0: Log is configured to *not* fsync() on all Append() calls
I20260927 03:32:09.570338 21571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15457badb08d4247801d70d30366e9c0: No bootstrap required, opened a new log
I20260927 03:32:09.571991 21571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15457badb08d4247801d70d30366e9c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15457badb08d4247801d70d30366e9c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36809 } }
I20260927 03:32:09.572131 21571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15457badb08d4247801d70d30366e9c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:32:09.572154 21571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15457badb08d4247801d70d30366e9c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15457badb08d4247801d70d30366e9c0, State: Initialized, Role: FOLLOWER
I20260927 03:32:09.572332 21571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15457badb08d4247801d70d30366e9c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15457badb08d4247801d70d30366e9c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36809 } }
I20260927 03:32:09.572418 21571 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15457badb08d4247801d70d30366e9c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:32:09.572443 21571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15457badb08d4247801d70d30366e9c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:32:09.572471 21571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15457badb08d4247801d70d30366e9c0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:32:09.573019 21571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15457badb08d4247801d70d30366e9c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15457badb08d4247801d70d30366e9c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36809 } }
I20260927 03:32:09.573150 21571 leader_election.cc:302] T 00000000000000000000000000000000 P 15457badb08d4247801d70d30366e9c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15457badb08d4247801d70d30366e9c0; no voters:
I20260927 03:32:09.573263 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 21503
I20260927 03:32:09.573328 21571 leader_election.cc:288] T 00000000000000000000000000000000 P 15457badb08d4247801d70d30366e9c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:32:09.573386 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:32:09.573395 21574 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15457badb08d4247801d70d30366e9c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:32:09.573589 21574 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15457badb08d4247801d70d30366e9c0 [term 1 LEADER]: Becoming Leader. State: Replica: 15457badb08d4247801d70d30366e9c0, State: Running, Role: LEADER
I20260927 03:32:09.573616 392 external_mini_cluster.cc:1428] Setting key cb86c750456cd8a86e7c0d02bc934770
I20260927 03:32:09.573652 21571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15457badb08d4247801d70d30366e9c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:32:09.573673 21574 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15457badb08d4247801d70d30366e9c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15457badb08d4247801d70d30366e9c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36809 } }
I20260927 03:32:09.574047 21574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15457badb08d4247801d70d30366e9c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15457badb08d4247801d70d30366e9c0. Latest consensus state: current_term: 1 leader_uuid: "15457badb08d4247801d70d30366e9c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15457badb08d4247801d70d30366e9c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36809 } } }
I20260927 03:32:09.574133 21574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15457badb08d4247801d70d30366e9c0 [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:09.574266 21575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15457badb08d4247801d70d30366e9c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15457badb08d4247801d70d30366e9c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15457badb08d4247801d70d30366e9c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36809 } } }
I20260927 03:32:09.574333 21575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15457badb08d4247801d70d30366e9c0 [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:09.574793 21579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:32:09.575619 21579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:32:09.576583 21581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:09.574301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:38863 (local address 127.0.0.1:36809)
0927 03:32:09.574763 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:09.574774 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:09.574816 (+ 42us) server_negotiation.cc:407] Connection header received
0927 03:32:09.574950 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:09.574957 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:09.575000 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:09.575153 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:09.575898 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:09.575903 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:09.575903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:32:09.575907 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:09.575946 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:32:09.575960 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:09.576111 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:09.576342 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 03:32:09.576389 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":348,"threads_started":1}
I20260927 03:32:09.577389 21579 catalog_manager.cc:1347] Generated new cluster ID: 0ad8f6f2aa3445f88269538f5b30f8c7
I20260927 03:32:09.577466 21579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:32:09.577649 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:32:09.580365 21581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:09.579211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:50735 (local address 127.0.0.1:36809)
0927 03:32:09.579241 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:09.579242 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:09.579368 (+ 126us) server_negotiation.cc:407] Connection header received
0927 03:32:09.579498 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:09.579499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:09.579503 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:09.579529 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:09.580160 (+ 631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:09.580161 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:09.580161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:32:09.580165 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:09.580178 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:32:09.580181 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:09.580206 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:09.580295 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 03:32:09.580307 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260927 03:32:09.590739 21579 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:32:09.590916 21579 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:32:09.593994 21579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15457badb08d4247801d70d30366e9c0: Generated new TSK 0
I20260927 03:32:09.594175 21579 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:32:09.598057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 21503
2026-09-27T03:32:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1288 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:32:09 dist-test-slave-j0hf krb5kdc[21602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:32:09 dist-test-slave-j0hf krb5kdc[21602](info): set up 2 sockets
Sep 27 03:32:09 dist-test-slave-j0hf krb5kdc[21602](info): commencing operation
krb5kdc: starting...
W20260927 03:32:10.627714 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:32:10 dist-test-slave-j0hf krb5kdc[21602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:32:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:32:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:32:10.740008 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38815
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:32:10.820631 21615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:32:10.820834 21615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:32:10.820858 21615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:32:10.822029 21615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:32:10.822072 21615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:32:10.822088 21615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:32:10.822100 21615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:32:10.822113 21615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:32:10.823661 21615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38815
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:32:10.823894 21615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:32:10.824138 21615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:32:10.828857 21620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:10.828857 21623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:10.828879 21621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:32:10.829298 21615 server_base.cc:1034] running on GCE node
I20260927 03:32:10.829577 21615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:32:10.829891 21615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:32:10.831067 21615 hybrid_clock.cc:648] HybridClock initialized: now 1790479930831040 us; error 36 us; skew 500 ppm
Sep 27 03:32:10 dist-test-slave-j0hf krb5kdc[21602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:32:10.838796 21615 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:32:10.839286 21615 webserver.cc:460] Webserver started at http://127.0.0.1:34349/ using document root <none> and password file <none>
I20260927 03:32:10.839643 21615 fs_manager.cc:359] Metadata directory not provided
I20260927 03:32:10.839730 21615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:32:10.839898 21615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:32:10.841017 21615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "4041ee52e410442ebb1caaf1c47015b9"
format_stamp: "Formatted at 2026-09-27 03:32:10 on dist-test-slave-j0hf"
server_key: "f2cfe6d252473c6fa4c4929d239338ef"
server_key_iv: "cb80babd6cd895f4fcf62438b3f8e680"
server_key_version: "encryptionkey@0"
I20260927 03:32:10.841416 21615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "4041ee52e410442ebb1caaf1c47015b9"
format_stamp: "Formatted at 2026-09-27 03:32:10 on dist-test-slave-j0hf"
server_key: "f2cfe6d252473c6fa4c4929d239338ef"
server_key_iv: "cb80babd6cd895f4fcf62438b3f8e680"
server_key_version: "encryptionkey@0"
I20260927 03:32:10.842861 21615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:32:10.843760 21630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:32:10.843935 21615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:32:10.844069 21615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "4041ee52e410442ebb1caaf1c47015b9"
format_stamp: "Formatted at 2026-09-27 03:32:10 on dist-test-slave-j0hf"
server_key: "f2cfe6d252473c6fa4c4929d239338ef"
server_key_iv: "cb80babd6cd895f4fcf62438b3f8e680"
server_key_version: "encryptionkey@0"
I20260927 03:32:10.844144 21615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:32:10.861322 21615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:32:10.863216 21615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:32:10.863397 21615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:32:10.867111 21615 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38815
I20260927 03:32:10.867172 21682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38815 every 8 connection(s)
I20260927 03:32:10.867596 21615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:32:10.868477 21683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:32:10.870718 21683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4041ee52e410442ebb1caaf1c47015b9: Bootstrap starting.
I20260927 03:32:10.871312 21683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4041ee52e410442ebb1caaf1c47015b9: Neither blocks nor log segments found. Creating new log.
I20260927 03:32:10.871634 21683 log.cc:824] T 00000000000000000000000000000000 P 4041ee52e410442ebb1caaf1c47015b9: Log is configured to *not* fsync() on all Append() calls
I20260927 03:32:10.872309 21683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4041ee52e410442ebb1caaf1c47015b9: No bootstrap required, opened a new log
I20260927 03:32:10.873878 21683 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4041ee52e410442ebb1caaf1c47015b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4041ee52e410442ebb1caaf1c47015b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38815 } }
I20260927 03:32:10.874013 21683 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4041ee52e410442ebb1caaf1c47015b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:32:10.874066 21683 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4041ee52e410442ebb1caaf1c47015b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4041ee52e410442ebb1caaf1c47015b9, State: Initialized, Role: FOLLOWER
I20260927 03:32:10.874219 21683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4041ee52e410442ebb1caaf1c47015b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4041ee52e410442ebb1caaf1c47015b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38815 } }
I20260927 03:32:10.874281 21683 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4041ee52e410442ebb1caaf1c47015b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:32:10.874302 21683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4041ee52e410442ebb1caaf1c47015b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:32:10.874362 21683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4041ee52e410442ebb1caaf1c47015b9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:32:10.874727 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 21615
I20260927 03:32:10.874846 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:32:10.875046 392 external_mini_cluster.cc:1428] Setting key d8e5ccf8786d16458eeeb8b709b912c5
I20260927 03:32:10.875104 21683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4041ee52e410442ebb1caaf1c47015b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4041ee52e410442ebb1caaf1c47015b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38815 } }
I20260927 03:32:10.875205 21683 leader_election.cc:302] T 00000000000000000000000000000000 P 4041ee52e410442ebb1caaf1c47015b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4041ee52e410442ebb1caaf1c47015b9; no voters:
I20260927 03:32:10.875383 21683 leader_election.cc:288] T 00000000000000000000000000000000 P 4041ee52e410442ebb1caaf1c47015b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:32:10.875407 21686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4041ee52e410442ebb1caaf1c47015b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:32:10.875620 21686 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4041ee52e410442ebb1caaf1c47015b9 [term 1 LEADER]: Becoming Leader. State: Replica: 4041ee52e410442ebb1caaf1c47015b9, State: Running, Role: LEADER
I20260927 03:32:10.875703 21686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4041ee52e410442ebb1caaf1c47015b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4041ee52e410442ebb1caaf1c47015b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38815 } }
I20260927 03:32:10.875730 21683 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4041ee52e410442ebb1caaf1c47015b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:32:10.876247 21686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4041ee52e410442ebb1caaf1c47015b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4041ee52e410442ebb1caaf1c47015b9. Latest consensus state: current_term: 1 leader_uuid: "4041ee52e410442ebb1caaf1c47015b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4041ee52e410442ebb1caaf1c47015b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38815 } } }
I20260927 03:32:10.876346 21686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4041ee52e410442ebb1caaf1c47015b9 [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:10.876257 21688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4041ee52e410442ebb1caaf1c47015b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4041ee52e410442ebb1caaf1c47015b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4041ee52e410442ebb1caaf1c47015b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38815 } } }
I20260927 03:32:10.876480 21688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4041ee52e410442ebb1caaf1c47015b9 [sys.catalog]: This master's current role is: LEADER
W20260927 03:32:10.878118 21704 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4041ee52e410442ebb1caaf1c47015b9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:32:10.878175 21704 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:32:10.878217 21692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:32:10.878353 21692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:32:10.879894 21692 catalog_manager.cc:1347] Generated new cluster ID: 397f60d8d56740e0b050d37cda97e875
I20260927 03:32:10.879947 21692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:32:10 dist-test-slave-j0hf krb5kdc[21602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 03:32:10.886564 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:10.875570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:49513 (local address 127.0.0.1:38815)
0927 03:32:10.875783 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:10.875793 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:10.875989 (+ 196us) server_negotiation.cc:407] Connection header received
0927 03:32:10.876127 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:10.876137 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:10.876199 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:10.876358 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:10.877407 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:10.878638 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:10.879745 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:10.880278 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:10.880396 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:10.882748 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:10.882752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:10.882752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:10.882764 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:10.884984 (+ 2220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:10.885386 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:10.885389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:10.885389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:10.885498 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:10.885733 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:10.885734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:10.885735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:10.885969 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:10.886156 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:10.886243 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 03:32:10.886343 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":80,"threads_started":1}
I20260927 03:32:10.887102 21692 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:32:10.887596 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:32:10.889304 21692 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:32:10.892483 21692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4041ee52e410442ebb1caaf1c47015b9: Generated new TSK 0
I20260927 03:32:10.892629 21692 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:32:10.897771 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:10.889053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:47293 (local address 127.0.0.1:38815)
0927 03:32:10.889107 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:10.889109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:10.891593 (+ 2484us) server_negotiation.cc:407] Connection header received
0927 03:32:10.891670 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:10.891671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:10.891682 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:10.891752 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:10.892347 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:10.892954 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:10.893671 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:10.893896 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:10.893967 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:10.895150 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:10.895151 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:10.895151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:10.895158 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:10.896612 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:10.896969 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:10.896970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:10.896970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:10.897032 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:10.897271 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:10.897273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:10.897273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:10.897390 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:10.897500 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:10.897617 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 03:32:10.897678 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 03:32:10.899600 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 21615
W20260927 03:32:10.903658 21600 connection.cc:535] client connection to 127.0.0.1:38815 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38815): Connection reset by peer (error 104)
2026-09-27T03:32:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1301 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:32:10 dist-test-slave-j0hf krb5kdc[21714](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:32:10 dist-test-slave-j0hf krb5kdc[21714](info): set up 2 sockets
Sep 27 03:32:10 dist-test-slave-j0hf krb5kdc[21714](info): commencing operation
krb5kdc: starting...
W20260927 03:32:11.943626 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:32:12 dist-test-slave-j0hf krb5kdc[21714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:32:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:32:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:32:12.058252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32945
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:32:12.139358 21727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:32:12.139707 21727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:32:12.139750 21727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:32:12.140952 21727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:32:12.141034 21727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:32:12.141060 21727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:32:12.141083 21727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:32:12.141111 21727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:32:12.142652 21727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32945
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:32:12.142958 21727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:32:12.143203 21727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:32:12.148793 21735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:12.148823 21732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:32:12.148972 21727 server_base.cc:1034] running on GCE node
W20260927 03:32:12.148824 21733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:32:12.149336 21727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:32:12.149675 21727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:32:12.150887 21727 hybrid_clock.cc:648] HybridClock initialized: now 1790479932150849 us; error 41 us; skew 500 ppm
Sep 27 03:32:12 dist-test-slave-j0hf krb5kdc[21714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:32:12.153769 21727 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:32:12.154206 21727 webserver.cc:460] Webserver started at http://127.0.0.1:33857/ using document root <none> and password file <none>
I20260927 03:32:12.154439 21727 fs_manager.cc:359] Metadata directory not provided
I20260927 03:32:12.154508 21727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:32:12.154675 21727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:32:12.155823 21727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "693ac1e680bf459e80ea058d3ed9069f"
format_stamp: "Formatted at 2026-09-27 03:32:12 on dist-test-slave-j0hf"
server_key: "5db7fb6540134ebfd96ca733aa09ae17"
server_key_iv: "cbf81e135fb9e8bb07deaf45a9461378"
server_key_version: "encryptionkey@0"
I20260927 03:32:12.156208 21727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "693ac1e680bf459e80ea058d3ed9069f"
format_stamp: "Formatted at 2026-09-27 03:32:12 on dist-test-slave-j0hf"
server_key: "5db7fb6540134ebfd96ca733aa09ae17"
server_key_iv: "cbf81e135fb9e8bb07deaf45a9461378"
server_key_version: "encryptionkey@0"
I20260927 03:32:12.157532 21727 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:32:12.158259 21742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:32:12.158540 21727 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:32:12.158672 21727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "693ac1e680bf459e80ea058d3ed9069f"
format_stamp: "Formatted at 2026-09-27 03:32:12 on dist-test-slave-j0hf"
server_key: "5db7fb6540134ebfd96ca733aa09ae17"
server_key_iv: "cbf81e135fb9e8bb07deaf45a9461378"
server_key_version: "encryptionkey@0"
I20260927 03:32:12.158762 21727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:32:12.168196 21727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:32:12.170051 21727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:32:12.170220 21727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:32:12.174270 21727 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32945
I20260927 03:32:12.174301 21794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32945 every 8 connection(s)
I20260927 03:32:12.174659 21727 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:32:12.175709 21795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:32:12.177942 21795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 693ac1e680bf459e80ea058d3ed9069f: Bootstrap starting.
I20260927 03:32:12.178534 21795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 693ac1e680bf459e80ea058d3ed9069f: Neither blocks nor log segments found. Creating new log.
I20260927 03:32:12.178822 21795 log.cc:824] T 00000000000000000000000000000000 P 693ac1e680bf459e80ea058d3ed9069f: Log is configured to *not* fsync() on all Append() calls
I20260927 03:32:12.179556 21795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 693ac1e680bf459e80ea058d3ed9069f: No bootstrap required, opened a new log
I20260927 03:32:12.181105 21795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 693ac1e680bf459e80ea058d3ed9069f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693ac1e680bf459e80ea058d3ed9069f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32945 } }
I20260927 03:32:12.181239 21795 raft_consensus.cc:374] T 00000000000000000000000000000000 P 693ac1e680bf459e80ea058d3ed9069f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:32:12.181262 21795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 693ac1e680bf459e80ea058d3ed9069f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 693ac1e680bf459e80ea058d3ed9069f, State: Initialized, Role: FOLLOWER
I20260927 03:32:12.181429 21795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 693ac1e680bf459e80ea058d3ed9069f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693ac1e680bf459e80ea058d3ed9069f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32945 } }
I20260927 03:32:12.181496 21795 raft_consensus.cc:388] T 00000000000000000000000000000000 P 693ac1e680bf459e80ea058d3ed9069f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:32:12.181519 21795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 693ac1e680bf459e80ea058d3ed9069f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:32:12.181586 21795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 693ac1e680bf459e80ea058d3ed9069f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:32:12.182107 21795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 693ac1e680bf459e80ea058d3ed9069f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693ac1e680bf459e80ea058d3ed9069f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32945 } }
I20260927 03:32:12.182219 21795 leader_election.cc:302] T 00000000000000000000000000000000 P 693ac1e680bf459e80ea058d3ed9069f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 693ac1e680bf459e80ea058d3ed9069f; no voters:
I20260927 03:32:12.182392 21795 leader_election.cc:288] T 00000000000000000000000000000000 P 693ac1e680bf459e80ea058d3ed9069f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:32:12.182420 21798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 693ac1e680bf459e80ea058d3ed9069f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:32:12.182590 21798 raft_consensus.cc:686] T 00000000000000000000000000000000 P 693ac1e680bf459e80ea058d3ed9069f [term 1 LEADER]: Becoming Leader. State: Replica: 693ac1e680bf459e80ea058d3ed9069f, State: Running, Role: LEADER
I20260927 03:32:12.182651 21795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 693ac1e680bf459e80ea058d3ed9069f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:32:12.182751 21798 consensus_queue.cc:237] T 00000000000000000000000000000000 P 693ac1e680bf459e80ea058d3ed9069f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693ac1e680bf459e80ea058d3ed9069f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32945 } }
I20260927 03:32:12.182986 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 21727
I20260927 03:32:12.183125 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:32:12.183360 392 external_mini_cluster.cc:1428] Setting key 779dd14f6a396495f3468d198023843d
I20260927 03:32:12.183959 21799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693ac1e680bf459e80ea058d3ed9069f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "693ac1e680bf459e80ea058d3ed9069f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693ac1e680bf459e80ea058d3ed9069f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32945 } } }
I20260927 03:32:12.183933 21800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693ac1e680bf459e80ea058d3ed9069f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 693ac1e680bf459e80ea058d3ed9069f. Latest consensus state: current_term: 1 leader_uuid: "693ac1e680bf459e80ea058d3ed9069f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "693ac1e680bf459e80ea058d3ed9069f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32945 } } }
I20260927 03:32:12.184038 21799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693ac1e680bf459e80ea058d3ed9069f [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:12.184047 21800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693ac1e680bf459e80ea058d3ed9069f [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:12.184259 21812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:32:12.184856 21812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:32:12.186331 21812 catalog_manager.cc:1347] Generated new cluster ID: f654a5c452414fc588f19310b4a59c35
I20260927 03:32:12.186380 21812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:32:12.190228 21810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:12.183969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:36891 (local address 127.0.0.1:32945)
0927 03:32:12.184343 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:12.184352 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:12.184630 (+ 278us) server_negotiation.cc:407] Connection header received
0927 03:32:12.184832 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:12.184839 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:12.184898 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:12.185044 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:12.185290 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:12.186600 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:12.187766 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:12.188415 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:12.188501 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:12.189472 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:12.189477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:12.189477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:32:12.189691 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:12.189729 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:32:12.189744 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:12.189805 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:12.189986 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 03:32:12.190015 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":97,"threads_started":1}
I20260927 03:32:12.192520 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:32:12.197597 21812 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:32:12.198088 21812 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:32:12.206636 21812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 693ac1e680bf459e80ea058d3ed9069f: Generated new TSK 0
I20260927 03:32:12.206768 21810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:12.194081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:35457 (local address 127.0.0.1:32945)
0927 03:32:12.194115 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:12.194116 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:12.194237 (+ 121us) server_negotiation.cc:407] Connection header received
0927 03:32:12.194267 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:12.194268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:12.194278 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:12.194349 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:12.194485 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:12.195260 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:12.196419 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:12.196730 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:12.196804 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:12.206366 (+ 9562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:12.206368 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:12.206369 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:32:12.206375 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:12.206394 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:32:12.206399 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:12.206484 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:12.206677 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 03:32:12.206690 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 03:32:12.206877 21812 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:32:12.212702 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 21727
W20260927 03:32:12.216529 21712 connection.cc:535] client connection to 127.0.0.1:32945 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32945): Connection reset by peer (error 104)
2026-09-27T03:32:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:32:12 dist-test-slave-j0hf krb5kdc[21826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:32:12 dist-test-slave-j0hf krb5kdc[21826](info): set up 2 sockets
Sep 27 03:32:12 dist-test-slave-j0hf krb5kdc[21826](info): commencing operation
krb5kdc: starting...
W20260927 03:32:13.243602 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:32:13 dist-test-slave-j0hf krb5kdc[21826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:32:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:32:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 03:32:13.361853 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43129
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:32:13.443826 21839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:32:13.444036 21839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:32:13.444060 21839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:32:13.445158 21839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:32:13.445191 21839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:32:13.445205 21839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:32:13.445218 21839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:32:13.445230 21839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:32:13.446606 21839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43129
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479675865103-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:32:13.446784 21839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:32:13.447053 21839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:32:13.451737 21844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:32:13.451862 21839 server_base.cc:1034] running on GCE node
W20260927 03:32:13.451802 21845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:13.451767 21847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:32:13.452265 21839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:32:13.452570 21839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:32:13.453748 21839 hybrid_clock.cc:648] HybridClock initialized: now 1790479933453729 us; error 35 us; skew 500 ppm
Sep 27 03:32:13 dist-test-slave-j0hf krb5kdc[21826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:32:13.456096 21839 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 03:32:13.456489 21839 webserver.cc:460] Webserver started at http://127.0.0.1:45639/ using document root <none> and password file <none>
I20260927 03:32:13.456729 21839 fs_manager.cc:359] Metadata directory not provided
I20260927 03:32:13.456795 21839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:32:13.456947 21839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:32:13.457964 21839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "6626f59910694420a457883ffc5bc1d0"
format_stamp: "Formatted at 2026-09-27 03:32:13 on dist-test-slave-j0hf"
server_key: "81cab0963fe8e8df36884c2ba13b4ee2"
server_key_iv: "7cedb874ab33613df6e34eee9b08d78d"
server_key_version: "encryptionkey@0"
I20260927 03:32:13.458313 21839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "6626f59910694420a457883ffc5bc1d0"
format_stamp: "Formatted at 2026-09-27 03:32:13 on dist-test-slave-j0hf"
server_key: "81cab0963fe8e8df36884c2ba13b4ee2"
server_key_iv: "7cedb874ab33613df6e34eee9b08d78d"
server_key_version: "encryptionkey@0"
I20260927 03:32:13.459690 21839 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:32:13.460376 21854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:32:13.460551 21839 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:32:13.460661 21839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "6626f59910694420a457883ffc5bc1d0"
format_stamp: "Formatted at 2026-09-27 03:32:13 on dist-test-slave-j0hf"
server_key: "81cab0963fe8e8df36884c2ba13b4ee2"
server_key_iv: "7cedb874ab33613df6e34eee9b08d78d"
server_key_version: "encryptionkey@0"
I20260927 03:32:13.460733 21839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:32:13.463979 21839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:32:13.465754 21839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:32:13.465904 21839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:32:13.469743 21839 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43129
I20260927 03:32:13.469851 21906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43129 every 8 connection(s)
I20260927 03:32:13.470135 21839 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:32:13.471123 21907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:32:13.473400 21907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6626f59910694420a457883ffc5bc1d0: Bootstrap starting.
I20260927 03:32:13.474220 21907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6626f59910694420a457883ffc5bc1d0: Neither blocks nor log segments found. Creating new log.
I20260927 03:32:13.474611 21907 log.cc:824] T 00000000000000000000000000000000 P 6626f59910694420a457883ffc5bc1d0: Log is configured to *not* fsync() on all Append() calls
I20260927 03:32:13.475385 21907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6626f59910694420a457883ffc5bc1d0: No bootstrap required, opened a new log
I20260927 03:32:13.477072 21907 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6626f59910694420a457883ffc5bc1d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6626f59910694420a457883ffc5bc1d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43129 } }
I20260927 03:32:13.477213 21907 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6626f59910694420a457883ffc5bc1d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:32:13.477260 21907 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6626f59910694420a457883ffc5bc1d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6626f59910694420a457883ffc5bc1d0, State: Initialized, Role: FOLLOWER
I20260927 03:32:13.477425 21907 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6626f59910694420a457883ffc5bc1d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6626f59910694420a457883ffc5bc1d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43129 } }
I20260927 03:32:13.477458 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 21839
I20260927 03:32:13.477516 21907 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6626f59910694420a457883ffc5bc1d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:32:13.477566 21907 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6626f59910694420a457883ffc5bc1d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:32:13.477586 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:32:13.477623 21907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6626f59910694420a457883ffc5bc1d0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:32:13.477782 392 external_mini_cluster.cc:1428] Setting key abe09abc15c2c2f51ca266018b1164c8
I20260927 03:32:13.478356 21907 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6626f59910694420a457883ffc5bc1d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6626f59910694420a457883ffc5bc1d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43129 } }
I20260927 03:32:13.478495 21907 leader_election.cc:302] T 00000000000000000000000000000000 P 6626f59910694420a457883ffc5bc1d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6626f59910694420a457883ffc5bc1d0; no voters:
I20260927 03:32:13.478677 21907 leader_election.cc:288] T 00000000000000000000000000000000 P 6626f59910694420a457883ffc5bc1d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:32:13.478700 21912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6626f59910694420a457883ffc5bc1d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:32:13.478799 21912 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6626f59910694420a457883ffc5bc1d0 [term 1 LEADER]: Becoming Leader. State: Replica: 6626f59910694420a457883ffc5bc1d0, State: Running, Role: LEADER
I20260927 03:32:13.478880 21912 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6626f59910694420a457883ffc5bc1d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6626f59910694420a457883ffc5bc1d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43129 } }
I20260927 03:32:13.478946 21907 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6626f59910694420a457883ffc5bc1d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:32:13.479207 21912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6626f59910694420a457883ffc5bc1d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6626f59910694420a457883ffc5bc1d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6626f59910694420a457883ffc5bc1d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43129 } } }
I20260927 03:32:13.479346 21912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6626f59910694420a457883ffc5bc1d0 [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:13.479525 21912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6626f59910694420a457883ffc5bc1d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6626f59910694420a457883ffc5bc1d0. Latest consensus state: current_term: 1 leader_uuid: "6626f59910694420a457883ffc5bc1d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6626f59910694420a457883ffc5bc1d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43129 } } }
I20260927 03:32:13.479584 21912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6626f59910694420a457883ffc5bc1d0 [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:13.479688 21916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:32:13.480629 21916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:32:13.481097 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:13.478280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:43225 (local address 127.0.0.1:43129)
0927 03:32:13.478450 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:13.478459 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:13.478492 (+ 33us) server_negotiation.cc:407] Connection header received
0927 03:32:13.478636 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:13.478643 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:13.478679 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:13.478822 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:13.480458 (+ 1636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:13.480462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:13.480462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:32:13.480466 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:13.480507 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:32:13.480520 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:13.480582 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:13.480852 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 03:32:13.480899 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260927 03:32:13.482228 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:32:13.482360 21916 catalog_manager.cc:1347] Generated new cluster ID: 95d70f91226441cf931c11d821d55c36
I20260927 03:32:13.482393 21916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:32:13.484928 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:13.483691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:39785 (local address 127.0.0.1:43129)
0927 03:32:13.483723 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:13.483725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:13.484000 (+ 275us) server_negotiation.cc:407] Connection header received
0927 03:32:13.484045 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:13.484046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:13.484051 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:13.484105 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:13.484705 (+ 600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:13.484706 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:13.484707 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:32:13.484711 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:13.484726 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:32:13.484730 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:13.484791 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:13.484847 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 03:32:13.484859 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 03:32:13.500336 21916 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:32:13.500554 21916 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:32:13.504204 21916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6626f59910694420a457883ffc5bc1d0: Generated new TSK 0
I20260927 03:32:13.504410 21916 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:32:13.510388 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 21839
2026-09-27T03:32:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:32:13 dist-test-slave-j0hf krb5kdc[21938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:32:13 dist-test-slave-j0hf krb5kdc[21938](info): set up 2 sockets
Sep 27 03:32:13 dist-test-slave-j0hf krb5kdc[21938](info): commencing operation
krb5kdc: starting...
W20260927 03:32:14.547623 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:32:14 dist-test-slave-j0hf krb5kdc[21938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:32:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:32:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:32:14.665884 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43283
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:32:14.747083 21951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:32:14.747344 21951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:32:14.747385 21951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:32:14.748620 21951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:32:14.748695 21951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:32:14.748723 21951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:32:14.748754 21951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:32:14.748776 21951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:32:14.750291 21951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43283
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:32:14.750573 21951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:32:14.750814 21951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:32:14.755765 21957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:32:14.755930 21951 server_base.cc:1034] running on GCE node
W20260927 03:32:14.755798 21959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:14.755798 21956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:32:14.756258 21951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:32:14.756553 21951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:32:14.757697 21951 hybrid_clock.cc:648] HybridClock initialized: now 1790479934757679 us; error 30 us; skew 500 ppm
Sep 27 03:32:14 dist-test-slave-j0hf krb5kdc[21938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:32:14.760372 21951 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:32:14.760855 21951 webserver.cc:460] Webserver started at http://127.0.98.62:42417/ using document root <none> and password file <none>
I20260927 03:32:14.761081 21951 fs_manager.cc:359] Metadata directory not provided
I20260927 03:32:14.761127 21951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:32:14.761276 21951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:32:14.762331 21951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "d3d3567891c64747af73ab9b4eab889e"
format_stamp: "Formatted at 2026-09-27 03:32:14 on dist-test-slave-j0hf"
server_key: "543df6e99827ac83386869fa6049b5cc"
server_key_iv: "064dcac94721c33af7f8c0966404bf7a"
server_key_version: "encryptionkey@0"
I20260927 03:32:14.762717 21951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "d3d3567891c64747af73ab9b4eab889e"
format_stamp: "Formatted at 2026-09-27 03:32:14 on dist-test-slave-j0hf"
server_key: "543df6e99827ac83386869fa6049b5cc"
server_key_iv: "064dcac94721c33af7f8c0966404bf7a"
server_key_version: "encryptionkey@0"
I20260927 03:32:14.764127 21951 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:32:14.764873 21966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:32:14.765067 21951 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:32:14.765180 21951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "d3d3567891c64747af73ab9b4eab889e"
format_stamp: "Formatted at 2026-09-27 03:32:14 on dist-test-slave-j0hf"
server_key: "543df6e99827ac83386869fa6049b5cc"
server_key_iv: "064dcac94721c33af7f8c0966404bf7a"
server_key_version: "encryptionkey@0"
I20260927 03:32:14.765257 21951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:32:14.774461 21951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:32:14.776870 21951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:32:14.777040 21951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:32:14.781199 21951 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43283
I20260927 03:32:14.781237 22018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43283 every 8 connection(s)
I20260927 03:32:14.781594 21951 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:32:14.782469 22019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:32:14.784868 22019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3d3567891c64747af73ab9b4eab889e: Bootstrap starting.
I20260927 03:32:14.785447 22019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3d3567891c64747af73ab9b4eab889e: Neither blocks nor log segments found. Creating new log.
I20260927 03:32:14.785737 22019 log.cc:824] T 00000000000000000000000000000000 P d3d3567891c64747af73ab9b4eab889e: Log is configured to *not* fsync() on all Append() calls
I20260927 03:32:14.786410 22019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3d3567891c64747af73ab9b4eab889e: No bootstrap required, opened a new log
I20260927 03:32:14.788054 22019 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3d3567891c64747af73ab9b4eab889e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d3567891c64747af73ab9b4eab889e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43283 } }
I20260927 03:32:14.788195 22019 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3d3567891c64747af73ab9b4eab889e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:32:14.788218 22019 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3d3567891c64747af73ab9b4eab889e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3d3567891c64747af73ab9b4eab889e, State: Initialized, Role: FOLLOWER
I20260927 03:32:14.788371 22019 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3d3567891c64747af73ab9b4eab889e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d3567891c64747af73ab9b4eab889e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43283 } }
I20260927 03:32:14.788441 22019 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3d3567891c64747af73ab9b4eab889e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:32:14.788462 22019 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3d3567891c64747af73ab9b4eab889e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:32:14.788512 22019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3d3567891c64747af73ab9b4eab889e [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:32:14.789054 22019 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3d3567891c64747af73ab9b4eab889e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d3567891c64747af73ab9b4eab889e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43283 } }
I20260927 03:32:14.789140 22019 leader_election.cc:302] T 00000000000000000000000000000000 P d3d3567891c64747af73ab9b4eab889e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3d3567891c64747af73ab9b4eab889e; no voters:
I20260927 03:32:14.789345 22019 leader_election.cc:288] T 00000000000000000000000000000000 P d3d3567891c64747af73ab9b4eab889e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:32:14.789433 22022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3d3567891c64747af73ab9b4eab889e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:32:14.789604 22022 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3d3567891c64747af73ab9b4eab889e [term 1 LEADER]: Becoming Leader. State: Replica: d3d3567891c64747af73ab9b4eab889e, State: Running, Role: LEADER
I20260927 03:32:14.789625 22019 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3d3567891c64747af73ab9b4eab889e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:32:14.789703 22022 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3d3567891c64747af73ab9b4eab889e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d3567891c64747af73ab9b4eab889e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43283 } }
I20260927 03:32:14.790138 22024 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3d3567891c64747af73ab9b4eab889e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3d3567891c64747af73ab9b4eab889e. Latest consensus state: current_term: 1 leader_uuid: "d3d3567891c64747af73ab9b4eab889e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d3567891c64747af73ab9b4eab889e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43283 } } }
I20260927 03:32:14.790257 22023 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3d3567891c64747af73ab9b4eab889e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3d3567891c64747af73ab9b4eab889e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d3567891c64747af73ab9b4eab889e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43283 } } }
I20260927 03:32:14.790328 22023 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3d3567891c64747af73ab9b4eab889e [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:14.790261 22024 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3d3567891c64747af73ab9b4eab889e [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:14.790556 22031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:32:14.790693 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 21951
I20260927 03:32:14.790827 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:32:14.791020 392 external_mini_cluster.cc:1428] Setting key 7e17dcc3b20d86a9124243d04a639fe6
I20260927 03:32:14.791098 22031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:32:14.792753 22031 catalog_manager.cc:1347] Generated new cluster ID: f24747a477884700ba2bc3fa5adb6e02
I20260927 03:32:14.792804 22031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:32:14 dist-test-slave-j0hf krb5kdc[21938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:32:14.801338 22041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:14.791497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41289 (local address 127.0.98.62:43283)
0927 03:32:14.791856 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:14.791865 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:14.791898 (+ 33us) server_negotiation.cc:407] Connection header received
0927 03:32:14.792012 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:14.792019 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:14.792074 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:14.792230 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:14.793059 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:14.793957 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:14.794774 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:14.795235 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:14.795355 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:14.797841 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:14.797846 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:14.797846 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:14.797859 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:14.799770 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:14.800169 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:14.800176 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:14.800176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:14.800271 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:14.800499 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:14.800500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:14.800500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:14.800756 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:14.800939 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:14.801026 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 03:32:14.801129 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":76,"threads_started":1}
I20260927 03:32:14.802292 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:32:14.809788 22041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:14.803713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42087 (local address 127.0.98.62:43283)
0927 03:32:14.803751 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:14.803752 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:14.803863 (+ 111us) server_negotiation.cc:407] Connection header received
0927 03:32:14.803931 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:14.803932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:14.803942 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:14.803997 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:14.804494 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:14.805008 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:14.805701 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:14.805923 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:14.805997 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:14.807195 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:14.807196 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:14.807197 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:14.807204 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:14.808612 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:14.808982 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:14.808983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:14.808983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:14.809041 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:14.809266 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:14.809267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:14.809267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:14.809376 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:14.809500 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:14.809607 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 03:32:14.809669 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:32:14.822103 22031 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:32:14.822696 22031 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:32:14.826889 22031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3d3567891c64747af73ab9b4eab889e: Generated new TSK 0
I20260927 03:32:14.827085 22031 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:32:14.831132 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 21951
W20260927 03:32:14.835415 21936 connection.cc:535] client connection to 127.0.98.62:43283 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:43283): Connection reset by peer (error 104)
2026-09-27T03:32:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:32:14 dist-test-slave-j0hf krb5kdc[22050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:32:14 dist-test-slave-j0hf krb5kdc[22050](info): set up 2 sockets
Sep 27 03:32:14 dist-test-slave-j0hf krb5kdc[22050](info): commencing operation
krb5kdc: starting...
W20260927 03:32:15.855628 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:32:15 dist-test-slave-j0hf krb5kdc[22050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:32:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:32:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:32:15.972400 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37993
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:32:16.054454 22063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:32:16.054653 22063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:32:16.054694 22063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:32:16.055863 22063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:32:16.055903 22063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:32:16.055917 22063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:32:16.055930 22063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:32:16.055989 22063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:32:16.057413 22063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37993
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:32:16.057672 22063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:32:16.057912 22063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:32:16.062724 22063 server_base.cc:1034] running on GCE node
W20260927 03:32:16.062703 22071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:16.062698 22068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:16.062698 22069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:32:16.063166 22063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:32:16.063409 22063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:32:16.064620 22063 hybrid_clock.cc:648] HybridClock initialized: now 1790479936064610 us; error 40 us; skew 500 ppm
Sep 27 03:32:16 dist-test-slave-j0hf krb5kdc[22050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:32:16.072992 22063 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:32:16.073489 22063 webserver.cc:460] Webserver started at http://127.0.98.62:43843/ using document root <none> and password file <none>
I20260927 03:32:16.073731 22063 fs_manager.cc:359] Metadata directory not provided
I20260927 03:32:16.073800 22063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:32:16.073958 22063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:32:16.075053 22063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "f8241de962744904b5683850bd685db3"
format_stamp: "Formatted at 2026-09-27 03:32:16 on dist-test-slave-j0hf"
server_key: "7f8cdd98601b9cc7ed0cb2a1826292e1"
server_key_iv: "9a077514bb643a9aef28e21a65b4e3f2"
server_key_version: "encryptionkey@0"
I20260927 03:32:16.075479 22063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "f8241de962744904b5683850bd685db3"
format_stamp: "Formatted at 2026-09-27 03:32:16 on dist-test-slave-j0hf"
server_key: "7f8cdd98601b9cc7ed0cb2a1826292e1"
server_key_iv: "9a077514bb643a9aef28e21a65b4e3f2"
server_key_version: "encryptionkey@0"
I20260927 03:32:16.076851 22063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:32:16.077646 22078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:32:16.077857 22063 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:32:16.078015 22063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "f8241de962744904b5683850bd685db3"
format_stamp: "Formatted at 2026-09-27 03:32:16 on dist-test-slave-j0hf"
server_key: "7f8cdd98601b9cc7ed0cb2a1826292e1"
server_key_iv: "9a077514bb643a9aef28e21a65b4e3f2"
server_key_version: "encryptionkey@0"
I20260927 03:32:16.078105 22063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:32:16.093427 22063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:32:16.095884 22063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:32:16.096081 22063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:32:16.100639 22063 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37993
I20260927 03:32:16.100692 22130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37993 every 8 connection(s)
I20260927 03:32:16.101079 22063 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:32:16.101958 22131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:32:16.104278 22131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8241de962744904b5683850bd685db3: Bootstrap starting.
I20260927 03:32:16.104878 22131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8241de962744904b5683850bd685db3: Neither blocks nor log segments found. Creating new log.
I20260927 03:32:16.105167 22131 log.cc:824] T 00000000000000000000000000000000 P f8241de962744904b5683850bd685db3: Log is configured to *not* fsync() on all Append() calls
I20260927 03:32:16.105880 22131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8241de962744904b5683850bd685db3: No bootstrap required, opened a new log
I20260927 03:32:16.107189 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 22063
I20260927 03:32:16.107311 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:32:16.107537 392 external_mini_cluster.cc:1428] Setting key 55a6f7b24a31b6edc726988ba848b8cb
I20260927 03:32:16.107654 22131 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8241de962744904b5683850bd685db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8241de962744904b5683850bd685db3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } }
I20260927 03:32:16.107817 22131 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8241de962744904b5683850bd685db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:32:16.107842 22131 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8241de962744904b5683850bd685db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8241de962744904b5683850bd685db3, State: Initialized, Role: FOLLOWER
I20260927 03:32:16.107996 22131 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8241de962744904b5683850bd685db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8241de962744904b5683850bd685db3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } }
I20260927 03:32:16.108135 22131 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8241de962744904b5683850bd685db3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:32:16.108194 22131 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8241de962744904b5683850bd685db3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:32:16.108237 22131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8241de962744904b5683850bd685db3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:32:16.109364 22131 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8241de962744904b5683850bd685db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8241de962744904b5683850bd685db3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } }
I20260927 03:32:16.109467 22131 leader_election.cc:302] T 00000000000000000000000000000000 P f8241de962744904b5683850bd685db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8241de962744904b5683850bd685db3; no voters:
I20260927 03:32:16.109614 22131 leader_election.cc:288] T 00000000000000000000000000000000 P f8241de962744904b5683850bd685db3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:32:16.109647 22136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8241de962744904b5683850bd685db3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:32:16.109798 22136 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8241de962744904b5683850bd685db3 [term 1 LEADER]: Becoming Leader. State: Replica: f8241de962744904b5683850bd685db3, State: Running, Role: LEADER
I20260927 03:32:16.109848 22131 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8241de962744904b5683850bd685db3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:32:16.109926 22136 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8241de962744904b5683850bd685db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8241de962744904b5683850bd685db3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } }
I20260927 03:32:16.110388 22138 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8241de962744904b5683850bd685db3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8241de962744904b5683850bd685db3. Latest consensus state: current_term: 1 leader_uuid: "f8241de962744904b5683850bd685db3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8241de962744904b5683850bd685db3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } }
I20260927 03:32:16.110407 22137 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8241de962744904b5683850bd685db3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8241de962744904b5683850bd685db3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8241de962744904b5683850bd685db3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } }
I20260927 03:32:16.110484 22138 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8241de962744904b5683850bd685db3 [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:16.110494 22137 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8241de962744904b5683850bd685db3 [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:16.110677 22145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:32:16.111208 22145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:32:16.113179 22145 catalog_manager.cc:1347] Generated new cluster ID: 038dd9ed7ca14430bce28f24dba95da9
I20260927 03:32:16.113233 22145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:32:16.113749 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:16.108073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37031 (local address 127.0.98.62:37993)
0927 03:32:16.108296 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:16.108304 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:16.108337 (+ 33us) server_negotiation.cc:407] Connection header received
0927 03:32:16.108458 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:16.108466 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:16.108518 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:16.108689 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:16.108874 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:16.110266 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:16.111601 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:16.112022 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:16.112145 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:16.112974 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:16.112978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:16.112978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:32:16.113191 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:16.113228 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:32:16.113243 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:16.113330 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:16.113518 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 03:32:16.113545 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":101,"threads_started":1}
I20260927 03:32:16.114636 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:32:16.118444 22145 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:32:16.118912 22145 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:32:16.120035 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:16.116174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49915 (local address 127.0.98.62:37993)
0927 03:32:16.116210 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:16.116212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:16.116377 (+ 165us) server_negotiation.cc:407] Connection header received
0927 03:32:16.116469 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:16.116470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:16.116492 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:16.116523 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:16.116727 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:16.117480 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:16.118541 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:16.118890 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:16.118934 (+ 44us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:16.119763 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:16.119765 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:16.119765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:32:16.119769 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:16.119783 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:32:16.119787 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:16.119820 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:16.119951 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 03:32:16.119963 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 03:32:16.125044 22145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8241de962744904b5683850bd685db3: Generated new TSK 0
I20260927 03:32:16.125227 22145 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:32:16.129133 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 22063
W20260927 03:32:16.133524 22048 connection.cc:535] client connection to 127.0.98.62:37993 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:37993): Connection reset by peer (error 104)
2026-09-27T03:32:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:32:16 dist-test-slave-j0hf krb5kdc[22162](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:32:16 dist-test-slave-j0hf krb5kdc[22162](info): set up 2 sockets
Sep 27 03:32:16 dist-test-slave-j0hf krb5kdc[22162](info): commencing operation
krb5kdc: starting...
W20260927 03:32:17.163836 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:32:17 dist-test-slave-j0hf krb5kdc[22162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:32:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:32:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:32:17.284202 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34285
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:32:17.364961 22175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:32:17.365228 22175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:32:17.365272 22175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:32:17.366464 22175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:32:17.366539 22175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:32:17.366559 22175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:32:17.366580 22175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:32:17.366609 22175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:32:17.368148 22175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34285
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:32:17.368439 22175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:32:17.368680 22175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:32:17.373590 22181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:17.373564 22180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:32:17.373739 22175 server_base.cc:1034] running on GCE node
W20260927 03:32:17.373572 22183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:32:17.374070 22175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:32:17.374387 22175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:32:17.375550 22175 hybrid_clock.cc:648] HybridClock initialized: now 1790479937375511 us; error 58 us; skew 500 ppm
Sep 27 03:32:17 dist-test-slave-j0hf krb5kdc[22162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:32:17.378031 22175 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:32:17.378480 22175 webserver.cc:460] Webserver started at http://127.0.98.62:36855/ using document root <none> and password file <none>
I20260927 03:32:17.378858 22175 fs_manager.cc:359] Metadata directory not provided
I20260927 03:32:17.378943 22175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:32:17.379077 22175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:32:17.380586 22175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "29062e16466d4319b31f6a78c1d42cb3"
format_stamp: "Formatted at 2026-09-27 03:32:17 on dist-test-slave-j0hf"
server_key: "02319173965169fb88de1dc3578ef2d0"
server_key_iv: "a46388d324a08cf9de7e6e2dac0bebce"
server_key_version: "encryptionkey@0"
I20260927 03:32:17.380992 22175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "29062e16466d4319b31f6a78c1d42cb3"
format_stamp: "Formatted at 2026-09-27 03:32:17 on dist-test-slave-j0hf"
server_key: "02319173965169fb88de1dc3578ef2d0"
server_key_iv: "a46388d324a08cf9de7e6e2dac0bebce"
server_key_version: "encryptionkey@0"
I20260927 03:32:17.382498 22175 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:32:17.383244 22190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:32:17.383415 22175 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:32:17.383548 22175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "29062e16466d4319b31f6a78c1d42cb3"
format_stamp: "Formatted at 2026-09-27 03:32:17 on dist-test-slave-j0hf"
server_key: "02319173965169fb88de1dc3578ef2d0"
server_key_iv: "a46388d324a08cf9de7e6e2dac0bebce"
server_key_version: "encryptionkey@0"
I20260927 03:32:17.383630 22175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:32:17.386828 22175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:32:17.389175 22175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:32:17.389326 22175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:32:17.393334 22175 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34285
I20260927 03:32:17.393365 22242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34285 every 8 connection(s)
I20260927 03:32:17.393680 22175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:32:17.394728 22243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:32:17.397010 22243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29062e16466d4319b31f6a78c1d42cb3: Bootstrap starting.
I20260927 03:32:17.397532 22243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29062e16466d4319b31f6a78c1d42cb3: Neither blocks nor log segments found. Creating new log.
I20260927 03:32:17.397775 22243 log.cc:824] T 00000000000000000000000000000000 P 29062e16466d4319b31f6a78c1d42cb3: Log is configured to *not* fsync() on all Append() calls
I20260927 03:32:17.398406 22243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29062e16466d4319b31f6a78c1d42cb3: No bootstrap required, opened a new log
I20260927 03:32:17.399248 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 22175
I20260927 03:32:17.399369 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:32:17.399576 392 external_mini_cluster.cc:1428] Setting key 281bbb59bc7b43d1a2f437e97da4d8fa
I20260927 03:32:17.400630 22243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29062e16466d4319b31f6a78c1d42cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29062e16466d4319b31f6a78c1d42cb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34285 } }
I20260927 03:32:17.400826 22243 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29062e16466d4319b31f6a78c1d42cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:32:17.400909 22243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29062e16466d4319b31f6a78c1d42cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29062e16466d4319b31f6a78c1d42cb3, State: Initialized, Role: FOLLOWER
I20260927 03:32:17.401075 22243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29062e16466d4319b31f6a78c1d42cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29062e16466d4319b31f6a78c1d42cb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34285 } }
I20260927 03:32:17.401175 22243 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29062e16466d4319b31f6a78c1d42cb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:32:17.401199 22243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29062e16466d4319b31f6a78c1d42cb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:32:17.401273 22243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29062e16466d4319b31f6a78c1d42cb3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:32:17.401907 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:17.400098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32985 (local address 127.0.98.62:34285)
0927 03:32:17.400275 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:17.400283 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:17.400308 (+ 25us) server_negotiation.cc:407] Connection header received
0927 03:32:17.400606 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:17.400612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:17.400639 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:17.400752 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:17.401453 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:17.401455 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:17.401456 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:32:17.401459 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:17.401498 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:32:17.401509 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:17.401563 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:17.401653 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 03:32:17.401704 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
I20260927 03:32:17.402405 22243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29062e16466d4319b31f6a78c1d42cb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29062e16466d4319b31f6a78c1d42cb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34285 } }
I20260927 03:32:17.402537 22243 leader_election.cc:302] T 00000000000000000000000000000000 P 29062e16466d4319b31f6a78c1d42cb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29062e16466d4319b31f6a78c1d42cb3; no voters:
I20260927 03:32:17.402734 22243 leader_election.cc:288] T 00000000000000000000000000000000 P 29062e16466d4319b31f6a78c1d42cb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:32:17.402779 22248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29062e16466d4319b31f6a78c1d42cb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:32:17.402798 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:32:17.402909 22248 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29062e16466d4319b31f6a78c1d42cb3 [term 1 LEADER]: Becoming Leader. State: Replica: 29062e16466d4319b31f6a78c1d42cb3, State: Running, Role: LEADER
I20260927 03:32:17.402983 22243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29062e16466d4319b31f6a78c1d42cb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:32:17.403041 22248 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29062e16466d4319b31f6a78c1d42cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29062e16466d4319b31f6a78c1d42cb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34285 } }
I20260927 03:32:17.403448 22249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29062e16466d4319b31f6a78c1d42cb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29062e16466d4319b31f6a78c1d42cb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29062e16466d4319b31f6a78c1d42cb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34285 } } }
I20260927 03:32:17.403585 22249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29062e16466d4319b31f6a78c1d42cb3 [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:17.403415 22248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29062e16466d4319b31f6a78c1d42cb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29062e16466d4319b31f6a78c1d42cb3. Latest consensus state: current_term: 1 leader_uuid: "29062e16466d4319b31f6a78c1d42cb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29062e16466d4319b31f6a78c1d42cb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34285 } } }
I20260927 03:32:17.403873 22248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29062e16466d4319b31f6a78c1d42cb3 [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:17.404230 22257 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:32:17.404736 22257 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:32:17.405632 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:17.404533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40705 (local address 127.0.98.62:34285)
0927 03:32:17.404567 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:17.404569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:17.404632 (+ 63us) server_negotiation.cc:407] Connection header received
0927 03:32:17.404733 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:17.404733 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:17.404738 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:17.404764 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:17.405396 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:17.405397 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:17.405397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:32:17.405401 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:17.405416 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:32:17.405420 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:17.405484 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:17.405553 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 03:32:17.405567 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:32:17.406271 22257 catalog_manager.cc:1347] Generated new cluster ID: b70c683b39f142168b8f4bf8acc99bd1
I20260927 03:32:17.406317 22257 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:32:17.417083 22257 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:32:17.417253 22257 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:32:17.420524 22257 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29062e16466d4319b31f6a78c1d42cb3: Generated new TSK 0
I20260927 03:32:17.420729 22257 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:32:17.427386 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 22175
2026-09-27T03:32:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:32:17 dist-test-slave-j0hf krb5kdc[22274](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:32:17 dist-test-slave-j0hf krb5kdc[22274](info): set up 2 sockets
Sep 27 03:32:17 dist-test-slave-j0hf krb5kdc[22274](info): commencing operation
krb5kdc: starting...
W20260927 03:32:18.447712 392 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:32:18 dist-test-slave-j0hf krb5kdc[22274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:32:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:32:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:32:18.561528 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43289
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:32:18.640767 22287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:32:18.640971 22287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:32:18.640995 22287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:32:18.642072 22287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:32:18.642105 22287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:32:18.642119 22287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:32:18.642131 22287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:32:18.642149 22287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:32:18.643544 22287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:32:18.643731 22287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:32:18.643900 22287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:32:18.648845 22295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:18.648861 22293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:18.648864 22292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:32:18.649104 22287 server_base.cc:1034] running on GCE node
I20260927 03:32:18.649367 22287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:32:18.649638 22287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:32:18.650801 22287 hybrid_clock.cc:648] HybridClock initialized: now 1790479938650795 us; error 39 us; skew 500 ppm
Sep 27 03:32:18 dist-test-slave-j0hf krb5kdc[22274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:32:18.653247 22287 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:32:18.653658 22287 webserver.cc:460] Webserver started at http://127.0.98.62:44609/ using document root <none> and password file <none>
I20260927 03:32:18.653921 22287 fs_manager.cc:359] Metadata directory not provided
I20260927 03:32:18.653987 22287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:32:18.654139 22287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:32:18.655184 22287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "bd90a404cfa644d6a4907a2d16440a76"
format_stamp: "Formatted at 2026-09-27 03:32:18 on dist-test-slave-j0hf"
server_key: "4778ab7d5620f9b76d08ce60da94d306"
server_key_iv: "408fbc9c553c48d9c797cc602ddb8cc8"
server_key_version: "encryptionkey@0"
I20260927 03:32:18.655575 22287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "bd90a404cfa644d6a4907a2d16440a76"
format_stamp: "Formatted at 2026-09-27 03:32:18 on dist-test-slave-j0hf"
server_key: "4778ab7d5620f9b76d08ce60da94d306"
server_key_iv: "408fbc9c553c48d9c797cc602ddb8cc8"
server_key_version: "encryptionkey@0"
I20260927 03:32:18.656857 22287 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:32:18.657528 22302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:32:18.657693 22287 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:32:18.657790 22287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "bd90a404cfa644d6a4907a2d16440a76"
format_stamp: "Formatted at 2026-09-27 03:32:18 on dist-test-slave-j0hf"
server_key: "4778ab7d5620f9b76d08ce60da94d306"
server_key_iv: "408fbc9c553c48d9c797cc602ddb8cc8"
server_key_version: "encryptionkey@0"
I20260927 03:32:18.657874 22287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:32:18.672765 22287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:32:18.674597 22287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:32:18.674779 22287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:32:18.678766 22287 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43289
I20260927 03:32:18.678798 22354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43289 every 8 connection(s)
I20260927 03:32:18.679136 22287 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:32:18.680040 22355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:32:18.682269 22355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd90a404cfa644d6a4907a2d16440a76: Bootstrap starting.
I20260927 03:32:18.682855 22355 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd90a404cfa644d6a4907a2d16440a76: Neither blocks nor log segments found. Creating new log.
I20260927 03:32:18.683141 22355 log.cc:824] T 00000000000000000000000000000000 P bd90a404cfa644d6a4907a2d16440a76: Log is configured to *not* fsync() on all Append() calls
I20260927 03:32:18.683880 22355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd90a404cfa644d6a4907a2d16440a76: No bootstrap required, opened a new log
I20260927 03:32:18.685451 22355 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd90a404cfa644d6a4907a2d16440a76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd90a404cfa644d6a4907a2d16440a76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43289 } }
I20260927 03:32:18.685590 22355 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd90a404cfa644d6a4907a2d16440a76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:32:18.685612 22355 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd90a404cfa644d6a4907a2d16440a76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd90a404cfa644d6a4907a2d16440a76, State: Initialized, Role: FOLLOWER
I20260927 03:32:18.685786 22355 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd90a404cfa644d6a4907a2d16440a76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd90a404cfa644d6a4907a2d16440a76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43289 } }
I20260927 03:32:18.685889 22355 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd90a404cfa644d6a4907a2d16440a76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:32:18.685928 22355 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd90a404cfa644d6a4907a2d16440a76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:32:18.685984 22355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd90a404cfa644d6a4907a2d16440a76 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:32:18.685997 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 22287
I20260927 03:32:18.686101 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:32:18.686312 392 external_mini_cluster.cc:1428] Setting key 6d5281577c0ad39d4722e44af0bef92c
I20260927 03:32:18.686700 22355 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd90a404cfa644d6a4907a2d16440a76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd90a404cfa644d6a4907a2d16440a76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43289 } }
I20260927 03:32:18.686827 22355 leader_election.cc:302] T 00000000000000000000000000000000 P bd90a404cfa644d6a4907a2d16440a76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd90a404cfa644d6a4907a2d16440a76; no voters:
I20260927 03:32:18.687004 22355 leader_election.cc:288] T 00000000000000000000000000000000 P bd90a404cfa644d6a4907a2d16440a76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:32:18.687050 22360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd90a404cfa644d6a4907a2d16440a76 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:32:18.687206 22360 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd90a404cfa644d6a4907a2d16440a76 [term 1 LEADER]: Becoming Leader. State: Replica: bd90a404cfa644d6a4907a2d16440a76, State: Running, Role: LEADER
I20260927 03:32:18.687310 22355 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd90a404cfa644d6a4907a2d16440a76 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:32:18.687374 22360 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd90a404cfa644d6a4907a2d16440a76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd90a404cfa644d6a4907a2d16440a76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43289 } }
I20260927 03:32:18.687846 22361 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd90a404cfa644d6a4907a2d16440a76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd90a404cfa644d6a4907a2d16440a76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd90a404cfa644d6a4907a2d16440a76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43289 } } }
I20260927 03:32:18.687929 22361 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd90a404cfa644d6a4907a2d16440a76 [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:18.688117 22362 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd90a404cfa644d6a4907a2d16440a76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd90a404cfa644d6a4907a2d16440a76. Latest consensus state: current_term: 1 leader_uuid: "bd90a404cfa644d6a4907a2d16440a76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd90a404cfa644d6a4907a2d16440a76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43289 } } }
I20260927 03:32:18.688349 22362 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd90a404cfa644d6a4907a2d16440a76 [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:18.688508 22364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:32:18.688982 22364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:32:18.690863 22364 catalog_manager.cc:1347] Generated new cluster ID: e4f23652fc9c49b1934395b9c1ce416c
I20260927 03:32:18.690914 22364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:32:18 dist-test-slave-j0hf krb5kdc[22274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:32:18.698153 22359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:18.686790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34069 (local address 127.0.98.62:43289)
0927 03:32:18.687010 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:18.687020 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:18.687057 (+ 37us) server_negotiation.cc:407] Connection header received
0927 03:32:18.687180 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:18.687188 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:18.687250 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:18.688259 (+ 1009us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:18.688362 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:18.689748 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:18.690810 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:18.691392 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:18.691519 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:18.694270 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:18.694274 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:18.694275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:18.694286 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:18.696291 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:18.696655 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:18.696658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:18.696659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:18.696760 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:18.696932 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:18.696933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:18.696933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:18.697216 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:18.697354 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:18.697802 (+ 448us) server_negotiation.cc:299] Negotiation successful
0927 03:32:18.697929 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":85,"threads_started":1}
I20260927 03:32:18.699133 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:32:18.714828 22359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:18.700514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39529 (local address 127.0.98.62:43289)
0927 03:32:18.700545 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:18.700547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:18.707623 (+ 7076us) server_negotiation.cc:407] Connection header received
0927 03:32:18.707706 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:18.707707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:18.707718 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:18.707766 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:18.708467 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:18.709238 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:18.710287 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:18.710586 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:18.710648 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:18.712144 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:18.712146 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:18.712146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:18.712153 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:18.713703 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:18.714084 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:18.714084 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:18.714085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:18.714135 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:18.714330 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:18.714330 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:18.714330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:18.714443 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:18.714544 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:18.714655 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 03:32:18.714733 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 03:32:18.735183 22364 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:32:18.735994 22364 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:32:18.745924 22364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd90a404cfa644d6a4907a2d16440a76: Generated new TSK 0
I20260927 03:32:18.746141 22364 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:32:18.758456 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 22287
2026-09-27T03:32:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:32:18 dist-test-slave-j0hf krb5kdc[22386](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:32:18 dist-test-slave-j0hf krb5kdc[22386](info): set up 2 sockets
Sep 27 03:32:18 dist-test-slave-j0hf krb5kdc[22386](info): commencing operation
krb5kdc: starting...
W20260927 03:32:19.791580 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:32:19 dist-test-slave-j0hf krb5kdc[22386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:32:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:32:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:32:19.904309 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33629
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:32:19.985611 22399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:32:19.985801 22399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:32:19.985823 22399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:32:19.986994 22399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:32:19.987036 22399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:32:19.987051 22399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:32:19.987063 22399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:32:19.987111 22399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:32:19.988581 22399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:32:19.988765 22399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:32:19.988996 22399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:32:19.993861 22404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:19.993885 22405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:19.993863 22407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:32:19.994149 22399 server_base.cc:1034] running on GCE node
I20260927 03:32:19.994407 22399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:32:19.994684 22399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:32:19.995867 22399 hybrid_clock.cc:648] HybridClock initialized: now 1790479939995868 us; error 44 us; skew 500 ppm
Sep 27 03:32:19 dist-test-slave-j0hf krb5kdc[22386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:32:19.998237 22399 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:32:19.998659 22399 webserver.cc:460] Webserver started at http://127.0.98.62:37781/ using document root <none> and password file <none>
I20260927 03:32:19.998884 22399 fs_manager.cc:359] Metadata directory not provided
I20260927 03:32:19.998951 22399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:32:19.999104 22399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:32:20.000288 22399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "12ce280f73b549d0a04df16752be7890"
format_stamp: "Formatted at 2026-09-27 03:32:19 on dist-test-slave-j0hf"
server_key: "2cfee01f56827f3029b56e8b04eee5af"
server_key_iv: "cf0d62ffb18e01f659c56626314498fc"
server_key_version: "encryptionkey@0"
I20260927 03:32:20.000658 22399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "12ce280f73b549d0a04df16752be7890"
format_stamp: "Formatted at 2026-09-27 03:32:19 on dist-test-slave-j0hf"
server_key: "2cfee01f56827f3029b56e8b04eee5af"
server_key_iv: "cf0d62ffb18e01f659c56626314498fc"
server_key_version: "encryptionkey@0"
I20260927 03:32:20.001992 22399 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:32:20.002732 22414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:32:20.002913 22399 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:32:20.003012 22399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "12ce280f73b549d0a04df16752be7890"
format_stamp: "Formatted at 2026-09-27 03:32:19 on dist-test-slave-j0hf"
server_key: "2cfee01f56827f3029b56e8b04eee5af"
server_key_iv: "cf0d62ffb18e01f659c56626314498fc"
server_key_version: "encryptionkey@0"
I20260927 03:32:20.003098 22399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:32:20.014067 22399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:32:20.016023 22399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:32:20.016189 22399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:32:20.020207 22399 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33629
I20260927 03:32:20.020234 22466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33629 every 8 connection(s)
I20260927 03:32:20.020587 22399 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:32:20.021459 22467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:32:20.023777 22467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12ce280f73b549d0a04df16752be7890: Bootstrap starting.
I20260927 03:32:20.024343 22467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12ce280f73b549d0a04df16752be7890: Neither blocks nor log segments found. Creating new log.
I20260927 03:32:20.024638 22467 log.cc:824] T 00000000000000000000000000000000 P 12ce280f73b549d0a04df16752be7890: Log is configured to *not* fsync() on all Append() calls
I20260927 03:32:20.025328 22467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12ce280f73b549d0a04df16752be7890: No bootstrap required, opened a new log
I20260927 03:32:20.026877 22467 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12ce280f73b549d0a04df16752be7890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ce280f73b549d0a04df16752be7890" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33629 } }
I20260927 03:32:20.027014 22467 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12ce280f73b549d0a04df16752be7890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:32:20.027072 22467 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12ce280f73b549d0a04df16752be7890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12ce280f73b549d0a04df16752be7890, State: Initialized, Role: FOLLOWER
I20260927 03:32:20.027249 22467 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12ce280f73b549d0a04df16752be7890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ce280f73b549d0a04df16752be7890" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33629 } }
I20260927 03:32:20.027313 22467 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12ce280f73b549d0a04df16752be7890 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:32:20.027334 22467 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12ce280f73b549d0a04df16752be7890 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:32:20.027405 22467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12ce280f73b549d0a04df16752be7890 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:32:20.027933 22467 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12ce280f73b549d0a04df16752be7890 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ce280f73b549d0a04df16752be7890" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33629 } }
I20260927 03:32:20.028040 22467 leader_election.cc:302] T 00000000000000000000000000000000 P 12ce280f73b549d0a04df16752be7890 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12ce280f73b549d0a04df16752be7890; no voters:
I20260927 03:32:20.028226 22467 leader_election.cc:288] T 00000000000000000000000000000000 P 12ce280f73b549d0a04df16752be7890 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:32:20.028285 22470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12ce280f73b549d0a04df16752be7890 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:32:20.028446 22470 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12ce280f73b549d0a04df16752be7890 [term 1 LEADER]: Becoming Leader. State: Replica: 12ce280f73b549d0a04df16752be7890, State: Running, Role: LEADER
I20260927 03:32:20.028497 22467 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12ce280f73b549d0a04df16752be7890 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:32:20.028539 22470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12ce280f73b549d0a04df16752be7890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ce280f73b549d0a04df16752be7890" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33629 } }
I20260927 03:32:20.028910 22472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12ce280f73b549d0a04df16752be7890 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12ce280f73b549d0a04df16752be7890. Latest consensus state: current_term: 1 leader_uuid: "12ce280f73b549d0a04df16752be7890" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ce280f73b549d0a04df16752be7890" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33629 } } }
I20260927 03:32:20.028995 22472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12ce280f73b549d0a04df16752be7890 [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:20.029042 22471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12ce280f73b549d0a04df16752be7890 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12ce280f73b549d0a04df16752be7890" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12ce280f73b549d0a04df16752be7890" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33629 } } }
I20260927 03:32:20.029110 22471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12ce280f73b549d0a04df16752be7890 [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:20.029223 22478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:32:20.029875 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 22399
I20260927 03:32:20.029990 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:32:20.030179 392 external_mini_cluster.cc:1428] Setting key 06d4ca357ca8551a039f44a12ec4cf85
I20260927 03:32:20.030452 22478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:32:20.031984 22478 catalog_manager.cc:1347] Generated new cluster ID: c9611ed2d0ae440498106440e2252f11
I20260927 03:32:20.032025 22478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:32:20.041306 22489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:20.030797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54865 (local address 127.0.98.62:33629)
0927 03:32:20.035960 (+ 5163us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:20.035969 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:20.036037 (+ 68us) server_negotiation.cc:407] Connection header received
0927 03:32:20.036148 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:20.036165 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:20.036218 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:20.036338 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:20.036634 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:20.038002 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:20.039173 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:20.039771 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:20.039865 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:20.040623 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:20.040626 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:20.040627 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:32:20.040837 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:20.040872 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:32:20.040885 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:20.040944 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:20.041031 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 03:32:20.041091 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5107,"thread_start_us":60,"threads_started":1}
I20260927 03:32:20.042213 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:32:20.046639 22478 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:32:20.047314 22478 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:32:20.047392 22489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:20.043736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38199 (local address 127.0.98.62:33629)
0927 03:32:20.043765 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:20.043767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:20.043922 (+ 155us) server_negotiation.cc:407] Connection header received
0927 03:32:20.043977 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:20.043977 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:20.043986 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:20.044037 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:20.044162 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:20.044860 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:20.045833 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:20.046132 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:20.046211 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:20.047116 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:20.047117 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:20.047118 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:32:20.047122 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:20.047134 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:32:20.047138 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:20.047187 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:20.047239 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 03:32:20.047262 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":15}
I20260927 03:32:20.052227 22478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12ce280f73b549d0a04df16752be7890: Generated new TSK 0
I20260927 03:32:20.052412 22478 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:32:20.054229 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 22399
2026-09-27T03:32:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:32:20 dist-test-slave-j0hf krb5kdc[22498](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:32:20 dist-test-slave-j0hf krb5kdc[22498](info): set up 2 sockets
Sep 27 03:32:20 dist-test-slave-j0hf krb5kdc[22498](info): commencing operation
krb5kdc: starting...
W20260927 03:32:21.087687 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:32:21 dist-test-slave-j0hf krb5kdc[22498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:32:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:32:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:32:21.205777 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34247
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:32:21.285511 22511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:32:21.285722 22511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:32:21.285745 22511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:32:21.286924 22511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:32:21.286971 22511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:32:21.286985 22511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:32:21.287030 22511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:32:21.287060 22511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:32:21.288543 22511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:32:21.288767 22511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:32:21.288987 22511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:32:21.293799 22516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:32:21.293823 22511 server_base.cc:1034] running on GCE node
W20260927 03:32:21.293823 22517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:21.293807 22519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:32:21.294179 22511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:32:21.294471 22511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:32:21.295641 22511 hybrid_clock.cc:648] HybridClock initialized: now 1790479941295624 us; error 56 us; skew 500 ppm
Sep 27 03:32:21 dist-test-slave-j0hf krb5kdc[22498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:32:21.305347 22511 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:32:21.305858 22511 webserver.cc:460] Webserver started at http://127.0.98.62:41821/ using document root <none> and password file <none>
I20260927 03:32:21.306129 22511 fs_manager.cc:359] Metadata directory not provided
I20260927 03:32:21.306205 22511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:32:21.306352 22511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:32:21.307487 22511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "09b60c5be0e74d468618d79e84b20726"
format_stamp: "Formatted at 2026-09-27 03:32:21 on dist-test-slave-j0hf"
server_key: "11e84d272eba35269b70e80a05f67d28"
server_key_iv: "e9a248d3cf96786a98022f021f5b003c"
server_key_version: "encryptionkey@0"
I20260927 03:32:21.307865 22511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "09b60c5be0e74d468618d79e84b20726"
format_stamp: "Formatted at 2026-09-27 03:32:21 on dist-test-slave-j0hf"
server_key: "11e84d272eba35269b70e80a05f67d28"
server_key_iv: "e9a248d3cf96786a98022f021f5b003c"
server_key_version: "encryptionkey@0"
I20260927 03:32:21.309163 22511 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:32:21.309893 22526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:32:21.310067 22511 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:32:21.310158 22511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "09b60c5be0e74d468618d79e84b20726"
format_stamp: "Formatted at 2026-09-27 03:32:21 on dist-test-slave-j0hf"
server_key: "11e84d272eba35269b70e80a05f67d28"
server_key_iv: "e9a248d3cf96786a98022f021f5b003c"
server_key_version: "encryptionkey@0"
I20260927 03:32:21.310240 22511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:32:21.313570 22511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:32:21.315362 22511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:32:21.315548 22511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:32:21.319339 22511 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34247
I20260927 03:32:21.319370 22578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34247 every 8 connection(s)
I20260927 03:32:21.319720 22511 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:32:21.320132 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 22511
I20260927 03:32:21.320258 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:32:21.320498 392 external_mini_cluster.cc:1428] Setting key 3bc2670d04901f0cb15ac2202fdc5702
I20260927 03:32:21.320945 22579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:32:21.322901 22582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:21.321038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55329 (local address 127.0.98.62:34247)
0927 03:32:21.321251 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:21.321259 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:21.321282 (+ 23us) server_negotiation.cc:407] Connection header received
0927 03:32:21.321682 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:21.321688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:21.321711 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:21.321823 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:21.322458 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:21.322461 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:21.322461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:32:21.322465 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:21.322499 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:32:21.322508 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:21.322562 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:21.322654 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 03:32:21.322713 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":107,"threads_started":1}
I20260927 03:32:21.324026 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:32:21.324422 22579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09b60c5be0e74d468618d79e84b20726: Bootstrap starting.
I20260927 03:32:21.324995 22579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09b60c5be0e74d468618d79e84b20726: Neither blocks nor log segments found. Creating new log.
I20260927 03:32:21.325301 22579 log.cc:824] T 00000000000000000000000000000000 P 09b60c5be0e74d468618d79e84b20726: Log is configured to *not* fsync() on all Append() calls
I20260927 03:32:21.326100 22579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09b60c5be0e74d468618d79e84b20726: No bootstrap required, opened a new log
I20260927 03:32:21.326498 22582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:21.325414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55111 (local address 127.0.98.62:34247)
0927 03:32:21.325455 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:21.325456 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:21.325590 (+ 134us) server_negotiation.cc:407] Connection header received
0927 03:32:21.325618 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:21.325619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:21.325624 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:21.325687 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:21.326227 (+ 540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:21.326228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:21.326229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:32:21.326234 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:21.326249 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:32:21.326253 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:21.326281 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:21.326387 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 03:32:21.326417 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:32:21.328159 22579 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09b60c5be0e74d468618d79e84b20726 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b60c5be0e74d468618d79e84b20726" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34247 } }
I20260927 03:32:21.328299 22579 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09b60c5be0e74d468618d79e84b20726 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:32:21.328323 22579 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09b60c5be0e74d468618d79e84b20726 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09b60c5be0e74d468618d79e84b20726, State: Initialized, Role: FOLLOWER
I20260927 03:32:21.328435 22579 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09b60c5be0e74d468618d79e84b20726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b60c5be0e74d468618d79e84b20726" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34247 } }
I20260927 03:32:21.328521 22579 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09b60c5be0e74d468618d79e84b20726 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:32:21.328542 22579 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09b60c5be0e74d468618d79e84b20726 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:32:21.328570 22579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09b60c5be0e74d468618d79e84b20726 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:32:21.329059 22579 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09b60c5be0e74d468618d79e84b20726 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b60c5be0e74d468618d79e84b20726" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34247 } }
I20260927 03:32:21.329142 22579 leader_election.cc:302] T 00000000000000000000000000000000 P 09b60c5be0e74d468618d79e84b20726 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09b60c5be0e74d468618d79e84b20726; no voters:
I20260927 03:32:21.329275 22579 leader_election.cc:288] T 00000000000000000000000000000000 P 09b60c5be0e74d468618d79e84b20726 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:32:21.329360 22589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09b60c5be0e74d468618d79e84b20726 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:32:21.329517 22589 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09b60c5be0e74d468618d79e84b20726 [term 1 LEADER]: Becoming Leader. State: Replica: 09b60c5be0e74d468618d79e84b20726, State: Running, Role: LEADER
I20260927 03:32:21.329535 22579 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09b60c5be0e74d468618d79e84b20726 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:32:21.329607 22589 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09b60c5be0e74d468618d79e84b20726 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b60c5be0e74d468618d79e84b20726" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34247 } }
I20260927 03:32:21.329988 22590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09b60c5be0e74d468618d79e84b20726 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09b60c5be0e74d468618d79e84b20726" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b60c5be0e74d468618d79e84b20726" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34247 } } }
I20260927 03:32:21.330030 22591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09b60c5be0e74d468618d79e84b20726 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09b60c5be0e74d468618d79e84b20726. Latest consensus state: current_term: 1 leader_uuid: "09b60c5be0e74d468618d79e84b20726" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b60c5be0e74d468618d79e84b20726" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34247 } } }
I20260927 03:32:21.330099 22591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09b60c5be0e74d468618d79e84b20726 [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:21.330091 22590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09b60c5be0e74d468618d79e84b20726 [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:21.330520 22598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:32:21.330974 22598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:32:21.332419 22598 catalog_manager.cc:1347] Generated new cluster ID: 1016596f1a724368afda2c84ab8438ce
I20260927 03:32:21.332476 22598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:32:21.353467 22598 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:32:21.353673 22598 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:32:21.361953 22598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09b60c5be0e74d468618d79e84b20726: Generated new TSK 0
I20260927 03:32:21.362159 22598 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:32:21.374598 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 22511
2026-09-27T03:32:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1320 ms)
[----------] 18 tests from AuthTokenIssuingTest (23602 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:32:21 dist-test-slave-j0hf krb5kdc[22610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:32:21 dist-test-slave-j0hf krb5kdc[22610](info): set up 2 sockets
Sep 27 03:32:21 dist-test-slave-j0hf krb5kdc[22610](info): commencing operation
krb5kdc: starting...
W20260927 03:32:22.419611 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:32:22 dist-test-slave-j0hf krb5kdc[22610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:32:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:32:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:32:22.537941 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41473
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41473,127.0.98.61:46485,127.0.98.60:46877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:32:22.619506 22623 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 03:32:22.619731 22623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:32:22.619750 22623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:32:22.619771 22623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:32:22.620863 22623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:32:22.620898 22623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:32:22.620913 22623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:32:22.620925 22623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:32:22.620937 22623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:32:22.622367 22623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41473,127.0.98.61:46485,127.0.98.60:46877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:32:22.622596 22623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:32:22.622778 22623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:32:22.627626 22629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:22.627626 22631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:22.627683 22628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:32:22.627803 22623 server_base.cc:1034] running on GCE node
I20260927 03:32:22.628170 22623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:32:22.628414 22623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:32:22.629549 22623 hybrid_clock.cc:648] HybridClock initialized: now 1790479942629539 us; error 27 us; skew 500 ppm
Sep 27 03:32:22 dist-test-slave-j0hf krb5kdc[22610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:32:22.632040 22623 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:32:22.632548 22623 webserver.cc:460] Webserver started at http://127.0.98.62:35253/ using document root <none> and password file <none>
I20260927 03:32:22.632798 22623 fs_manager.cc:359] Metadata directory not provided
I20260927 03:32:22.632866 22623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:32:22.633018 22623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:32:22.634052 22623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "68cac1ec7efb4c4d8189c618cd53daed"
format_stamp: "Formatted at 2026-09-27 03:32:22 on dist-test-slave-j0hf"
server_key: "524e591f271ed3575af06f80628ad718"
server_key_iv: "52d4d5541b54d62f3fc0e8f8909f53fd"
server_key_version: "encryptionkey@0"
I20260927 03:32:22.634421 22623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "68cac1ec7efb4c4d8189c618cd53daed"
format_stamp: "Formatted at 2026-09-27 03:32:22 on dist-test-slave-j0hf"
server_key: "524e591f271ed3575af06f80628ad718"
server_key_iv: "52d4d5541b54d62f3fc0e8f8909f53fd"
server_key_version: "encryptionkey@0"
I20260927 03:32:22.635783 22623 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:32:22.636488 22638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:32:22.636655 22623 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:32:22.636762 22623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "68cac1ec7efb4c4d8189c618cd53daed"
format_stamp: "Formatted at 2026-09-27 03:32:22 on dist-test-slave-j0hf"
server_key: "524e591f271ed3575af06f80628ad718"
server_key_iv: "52d4d5541b54d62f3fc0e8f8909f53fd"
server_key_version: "encryptionkey@0"
I20260927 03:32:22.636835 22623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:32:22.658902 22623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:32:22.660878 22623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:32:22.661041 22623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:32:22.665225 22623 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41473
I20260927 03:32:22.665258 22690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41473 every 8 connection(s)
I20260927 03:32:22.665616 22623 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:32:22.666649 22691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:32:22.668256 22691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41473 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:32:22.673138 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 22623
I20260927 03:32:22.673249 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:32:22.673418 392 external_mini_cluster.cc:1428] Setting key 786473350d34f97d70da45aa48a0fd32
Sep 27 03:32:22 dist-test-slave-j0hf krb5kdc[22610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:32:22.678462 22694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.668968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42514 (local address 127.0.98.62:41473)
0927 03:32:22.669142 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:22.669152 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:22.669276 (+ 124us) server_negotiation.cc:407] Connection header received
0927 03:32:22.669827 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:22.669830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:22.669856 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:22.669951 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:22.670479 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.671304 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.672242 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.672534 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.672680 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:22.675236 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:22.675243 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:22.675243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:22.675250 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:22.676915 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:22.677364 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:22.677366 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:22.677367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:22.677441 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:22.677696 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:22.677697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:22.677697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:22.677889 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:22.677949 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:22.678186 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 03:32:22.678309 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":143,"thread_start_us":72,"threads_started":1}
I20260927 03:32:22.678579 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.668934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41473 (local address 127.0.0.1:42514)
0927 03:32:22.669133 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:22.669175 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:32:22.669670 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:32:22.669956 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:32:22.669959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:32:22.670229 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:32:22.670373 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:22.670378 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.671398 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.671400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:22.672141 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:22.672143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.672665 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.672666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:22.672725 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:22.673296 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:32:22.673306 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:32:22.675143 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:32:22.677002 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:22.677010 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:22.677012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:22.677280 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:22.677510 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:22.677510 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:22.677511 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:22.677628 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:22.677960 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:32:22.677967 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:32:22.678052 (+ 85us) client_negotiation.cc:769] Sending connection context
0927 03:32:22.678164 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 03:32:22.678398 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":63,"thread_start_us":80,"threads_started":1}
I20260927 03:32:22.679327 22691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:32:22.679764 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.679618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46485 (local address 127.0.0.1:33078)
0927 03:32:22.679673 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:22.679733 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 03:32:22.679916 22639 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46485: connect: Connection refused (error 111)
I20260927 03:32:22.680334 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.680225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46485 (local address 127.0.0.1:33088)
0927 03:32:22.680282 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:22.680311 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 03:32:22.680451 22691 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46485: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46485: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 03:32:22.696307 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46485
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41473,127.0.98.61:46485,127.0.98.60:46877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:32:22.729712 22691 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } attempt: 1
I20260927 03:32:22.730325 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.730154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46485 (local address 127.0.0.1:33102)
0927 03:32:22.730231 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:22.730285 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 03:32:22.730786 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.730677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46485 (local address 127.0.0.1:33114)
0927 03:32:22.730730 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:22.730759 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 03:32:22.730932 22691 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46485: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46485: connect: Connection refused (error 111)
W20260927 03:32:22.776777 22698 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 03:32:22.776993 22698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:32:22.777010 22698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:32:22.777030 22698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:32:22.778191 22698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:32:22.778241 22698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:32:22.778256 22698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:32:22.778268 22698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:32:22.778331 22698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:32:22.779860 22698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41473,127.0.98.61:46485,127.0.98.60:46877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:32:22.780112 22698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:32:22.780360 22698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:32:22.785041 22703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:22.785028 22706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:22.785017 22704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:32:22.785243 22698 server_base.cc:1034] running on GCE node
I20260927 03:32:22.785548 22698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:32:22.785852 22698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:32:22.787005 22698 hybrid_clock.cc:648] HybridClock initialized: now 1790479942786996 us; error 32 us; skew 500 ppm
Sep 27 03:32:22 dist-test-slave-j0hf krb5kdc[22610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:32:22.789417 22698 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 03:32:22.789897 22698 webserver.cc:460] Webserver started at http://127.0.98.61:34921/ using document root <none> and password file <none>
I20260927 03:32:22.790117 22698 fs_manager.cc:359] Metadata directory not provided
I20260927 03:32:22.790185 22698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:32:22.790350 22698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:32:22.791363 22698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-1/data/instance:
uuid: "3c777c75e2c147bf9ae2b115a6ccea87"
format_stamp: "Formatted at 2026-09-27 03:32:22 on dist-test-slave-j0hf"
server_key: "81ecef7be9502ce68b19c94d1ace6835"
server_key_iv: "6ad16f6e4575eefda7971488af915c82"
server_key_version: "encryptionkey@0"
I20260927 03:32:22.791787 22698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-1/wal/instance:
uuid: "3c777c75e2c147bf9ae2b115a6ccea87"
format_stamp: "Formatted at 2026-09-27 03:32:22 on dist-test-slave-j0hf"
server_key: "81ecef7be9502ce68b19c94d1ace6835"
server_key_iv: "6ad16f6e4575eefda7971488af915c82"
server_key_version: "encryptionkey@0"
I20260927 03:32:22.793231 22698 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:32:22.794046 22713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:32:22.794245 22698 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:32:22.794343 22698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-1/wal
uuid: "3c777c75e2c147bf9ae2b115a6ccea87"
format_stamp: "Formatted at 2026-09-27 03:32:22 on dist-test-slave-j0hf"
server_key: "81ecef7be9502ce68b19c94d1ace6835"
server_key_iv: "6ad16f6e4575eefda7971488af915c82"
server_key_version: "encryptionkey@0"
I20260927 03:32:22.794430 22698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:32:22.799103 22691 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } attempt: 2
I20260927 03:32:22.799506 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.799359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46485 (local address 127.0.0.1:33126)
0927 03:32:22.799419 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:22.799473 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 03:32:22.799861 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.799772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46485 (local address 127.0.0.1:33134)
0927 03:32:22.799815 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:22.799840 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 03:32:22.799969 22691 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46485: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46485: connect: Connection refused (error 111)
I20260927 03:32:22.803988 22698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:32:22.805754 22698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:32:22.805907 22698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:32:22.809906 22698 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46485
I20260927 03:32:22.809935 22765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46485 every 8 connection(s)
I20260927 03:32:22.810267 22698 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-1/data/info.pb
I20260927 03:32:22.810837 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 22698
I20260927 03:32:22.810913 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-1/wal/instance
I20260927 03:32:22.811090 392 external_mini_cluster.cc:1428] Setting key abc6c551c37a06cca133e36730e4421f
I20260927 03:32:22.811303 22766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:32:22.812809 22766 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41473 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 03:32:22 dist-test-slave-j0hf krb5kdc[22610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:32:22.824512 22694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.813502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42528 (local address 127.0.98.62:41473)
0927 03:32:22.813540 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:22.813542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:22.813751 (+ 209us) server_negotiation.cc:407] Connection header received
0927 03:32:22.814361 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:22.814363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:22.814377 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:22.814429 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:22.815046 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.815603 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.816571 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.816793 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.816891 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:22.821586 (+ 4695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:22.821588 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:22.821588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:22.821603 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:22.823008 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:22.823455 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:22.823456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:22.823457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:22.823515 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:22.823798 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:22.823799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:22.823800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:22.823947 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:22.824017 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:22.824376 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 03:32:22.824445 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:32:22.824856 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.813479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41473 (local address 127.0.0.1:42528)
0927 03:32:22.813637 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:22.813668 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 03:32:22.814259 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:32:22.814505 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:32:22.814509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:32:22.814771 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:32:22.814952 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:22.814956 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.815705 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.815706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:22.816478 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:22.816480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.816912 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.816914 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:22.817036 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:22.817550 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:32:22.817558 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:32:22.821470 (+ 3912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:32:22.823105 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:22.823114 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:22.823116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:22.823353 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:22.823605 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:22.823606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:22.823606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:22.823723 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:22.824030 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:32:22.824037 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:32:22.824219 (+ 182us) client_negotiation.cc:769] Sending connection context
0927 03:32:22.824356 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 03:32:22.824547 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260927 03:32:22.825495 22766 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:32:22 dist-test-slave-j0hf krb5kdc[22610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 03:32:22.833295 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.825778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46485 (local address 127.0.0.1:33148)
0927 03:32:22.825835 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:22.825848 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:32:22.825892 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:32:22.826147 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:32:22.826148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:32:22.826301 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:32:22.826367 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:22.826369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.827196 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.827197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:22.828105 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:22.828108 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.828548 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.828549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:22.828606 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:22.828983 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:32:22.828988 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:32:22.830659 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:32:22.832268 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:22.832269 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:22.832269 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:22.832425 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:22.832653 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:22.832654 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:22.832655 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:22.832721 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:22.833018 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:32:22.833019 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:32:22.833073 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 03:32:22.833146 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 03:32:22.833231 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 03:32:22.833426 22771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.825829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33148 (local address 127.0.98.61:46485)
0927 03:32:22.825980 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:22.825985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:22.826002 (+ 17us) server_negotiation.cc:407] Connection header received
0927 03:32:22.826031 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:22.826033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:22.826044 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:22.826123 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:22.826444 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.827113 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.828182 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.828474 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.828533 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:22.830766 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:22.830773 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:22.830773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:22.830779 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:22.832196 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:22.832521 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:22.832523 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:22.832524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:22.832575 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:22.832808 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:22.832809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:22.832810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:22.832956 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:22.833019 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:22.833201 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 03:32:22.833279 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260927 03:32:22.833998 22766 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46877 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:32:22.834340 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.834221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46877 (local address 127.0.0.1:36288)
0927 03:32:22.834269 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:22.834314 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 03:32:22.834475 22715 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46877: connect: Connection refused (error 111)
I20260927 03:32:22.834877 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.834776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46877 (local address 127.0.0.1:36290)
0927 03:32:22.834826 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:22.834850 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 03:32:22.835000 22766 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46877: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46877: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 03:32:22.836481 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46877
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41473,127.0.98.61:46485,127.0.98.60:46877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:32:22.884217 22766 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46877 } attempt: 1
I20260927 03:32:22.884703 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.884572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46877 (local address 127.0.0.1:36306)
0927 03:32:22.884636 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:22.884676 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 03:32:22.885071 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.884980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46877 (local address 127.0.0.1:36316)
0927 03:32:22.885023 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:22.885046 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 03:32:22.885154 22766 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46877: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46877: connect: Connection refused (error 111)
I20260927 03:32:22.891132 22691 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } attempt: 3
Sep 27 03:32:22 dist-test-slave-j0hf krb5kdc[22610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 03:32:22.898793 22771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.891504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33154 (local address 127.0.98.61:46485)
0927 03:32:22.891550 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:22.891551 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:22.891600 (+ 49us) server_negotiation.cc:407] Connection header received
0927 03:32:22.891683 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:22.891684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:22.891694 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:22.891741 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:22.892108 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.892595 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.893628 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.893835 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.893889 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:22.895894 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:22.895896 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:22.895896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:22.895903 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:22.897544 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:22.897929 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:22.897930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:22.897931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:22.898000 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:22.898238 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:22.898240 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:22.898240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:22.898392 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:22.898468 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:22.898639 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 03:32:22.898709 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 03:32:22.898797 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.891479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46485 (local address 127.0.0.1:33154)
0927 03:32:22.891521 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:22.891534 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:32:22.891620 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:32:22.891766 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:32:22.891768 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:32:22.891956 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:32:22.892031 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:22.892033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.892683 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.892684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:22.893538 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:22.893541 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.893908 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.893909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:22.893990 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:22.894408 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:32:22.894413 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:32:22.895806 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:32:22.897636 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:22.897638 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:22.897638 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:22.897847 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:22.898080 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:22.898081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:22.898081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:22.898157 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:22.898464 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:32:22.898465 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:32:22.898531 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 03:32:22.898597 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 03:32:22.898696 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260927 03:32:22.899219 22691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46877 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:32:22.899571 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.899482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46877 (local address 127.0.0.1:36324)
0927 03:32:22.899520 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:22.899548 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 03:32:22.899889 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.899836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46877 (local address 127.0.0.1:36330)
0927 03:32:22.899859 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:22.899877 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260927 03:32:22.899988 22691 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46877: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46877: connect: Connection refused (error 111)
W20260927 03:32:22.918460 22773 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 03:32:22.918659 22773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:32:22.918677 22773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:32:22.918696 22773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:32:22.919795 22773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:32:22.919832 22773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:32:22.919847 22773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:32:22.919859 22773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:32:22.919871 22773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:32:22.921401 22773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41473,127.0.98.61:46485,127.0.98.60:46877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:32:22.921584 22773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:32:22.921743 22773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:32:22.926332 22781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:22.926357 22779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:22.926400 22778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:32:22.926544 22773 server_base.cc:1034] running on GCE node
I20260927 03:32:22.926868 22773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:32:22.927157 22773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:32:22.928318 22773 hybrid_clock.cc:648] HybridClock initialized: now 1790479942928295 us; error 34 us; skew 500 ppm
Sep 27 03:32:22 dist-test-slave-j0hf krb5kdc[22610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:32:22.930729 22773 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 03:32:22.931151 22773 webserver.cc:460] Webserver started at http://127.0.98.60:42289/ using document root <none> and password file <none>
I20260927 03:32:22.931114 22691 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46877 } attempt: 1
I20260927 03:32:22.931481 22773 fs_manager.cc:359] Metadata directory not provided
I20260927 03:32:22.931555 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.931408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46877 (local address 127.0.0.1:36332)
0927 03:32:22.931494 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:22.931528 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 03:32:22.931568 22773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:32:22.931701 22773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:32:22.931892 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.931815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46877 (local address 127.0.0.1:36338)
0927 03:32:22.931846 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:22.931875 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260927 03:32:22.931996 22691 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46877: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46877: connect: Connection refused (error 111)
I20260927 03:32:22.932828 22773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-2/data/instance:
uuid: "234b8abf0e394e63bb30ab7d9c64ebd2"
format_stamp: "Formatted at 2026-09-27 03:32:22 on dist-test-slave-j0hf"
server_key: "c6687b4092a23f706afb2328da8fe23d"
server_key_iv: "766b9f238c8490c02684fbf14f0994e1"
server_key_version: "encryptionkey@0"
I20260927 03:32:22.933265 22773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-2/wal/instance:
uuid: "234b8abf0e394e63bb30ab7d9c64ebd2"
format_stamp: "Formatted at 2026-09-27 03:32:22 on dist-test-slave-j0hf"
server_key: "c6687b4092a23f706afb2328da8fe23d"
server_key_iv: "766b9f238c8490c02684fbf14f0994e1"
server_key_version: "encryptionkey@0"
I20260927 03:32:22.934589 22773 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:32:22.935280 22788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:32:22.935496 22773 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:32:22.935588 22773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-2/wal
uuid: "234b8abf0e394e63bb30ab7d9c64ebd2"
format_stamp: "Formatted at 2026-09-27 03:32:22 on dist-test-slave-j0hf"
server_key: "c6687b4092a23f706afb2328da8fe23d"
server_key_iv: "766b9f238c8490c02684fbf14f0994e1"
server_key_version: "encryptionkey@0"
I20260927 03:32:22.935659 22773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:32:22.948253 22773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:32:22.950459 22773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:32:22.950629 22773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:32:22.953299 22766 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46877 } attempt: 2
I20260927 03:32:22.953734 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.953597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46877 (local address 127.0.0.1:36354)
0927 03:32:22.953668 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:22.953703 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 03:32:22.954160 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.954039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46877 (local address 127.0.0.1:36370)
0927 03:32:22.954068 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:22.954130 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260927 03:32:22.954268 22766 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46877: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46877: connect: Connection refused (error 111)
I20260927 03:32:22.954872 22773 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46877
I20260927 03:32:22.954901 22840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46877 every 8 connection(s)
I20260927 03:32:22.955240 22773 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-2/data/info.pb
I20260927 03:32:22.956272 22841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:32:22.957769 22841 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41473 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:32:22.960954 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 22773
I20260927 03:32:22.961035 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790479675865103-392-0/minicluster-data/master-2/wal/instance
I20260927 03:32:22.961202 392 external_mini_cluster.cc:1428] Setting key ec42516ab888155a40d10902f0a5c817
Sep 27 03:32:22 dist-test-slave-j0hf krb5kdc[22610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:32:22.966771 22694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.958465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42542 (local address 127.0.98.62:41473)
0927 03:32:22.958515 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:22.958516 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:22.958680 (+ 164us) server_negotiation.cc:407] Connection header received
0927 03:32:22.959106 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:22.959106 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:22.959115 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:22.959166 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:22.959787 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.960241 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.961272 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.961504 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.961596 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:22.963786 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:22.963787 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:22.963788 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:22.963795 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:22.965447 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:22.965817 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:22.965818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:22.965818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:22.965881 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:22.966107 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:22.966108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:22.966109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:22.966264 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:22.966328 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:22.966638 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 03:32:22.966700 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Sep 27 03:32:22 dist-test-slave-j0hf krb5kdc[22610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 03:32:22.967006 22843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.958396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41473 (local address 127.0.0.1:42542)
0927 03:32:22.958547 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:22.958589 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:32:22.959026 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:32:22.959224 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:32:22.959227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:32:22.959522 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:32:22.959717 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:22.959722 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.960315 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.960316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:22.961186 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:22.961188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.961696 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.961697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:22.961836 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:22.962338 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:32:22.962346 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:32:22.963697 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:32:22.965522 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:22.965527 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:22.965528 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:22.965741 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:22.965951 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:22.965951 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:22.965952 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:22.966042 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:22.966335 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:32:22.966341 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:32:22.966495 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 03:32:22.966620 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 03:32:22.966772 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
I20260927 03:32:22.967751 22841 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:32:22.970662 22845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.961597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36372 (local address 127.0.98.60:46877)
0927 03:32:22.961888 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:22.961892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:22.961906 (+ 14us) server_negotiation.cc:407] Connection header received
0927 03:32:22.961923 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:22.961924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:22.961938 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:22.962014 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:22.962647 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.963307 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.964350 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.964729 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.964794 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:22.967399 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:22.967404 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:22.967404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:22.967554 (+ 150us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:22.969478 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:22.969845 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:22.969847 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:22.969848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:22.969896 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:22.970088 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:22.970088 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:22.970088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:22.970256 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:22.970383 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:22.970453 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 03:32:22.970533 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":83,"threads_started":1}
Sep 27 03:32:22 dist-test-slave-j0hf krb5kdc[22610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 03:32:22.974931 22771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.968083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33160 (local address 127.0.98.61:46485)
0927 03:32:22.968153 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:22.968155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:22.968163 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:32:22.968188 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:22.968189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:22.968199 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:22.968267 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:22.968581 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.969299 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.970157 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.970467 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.970534 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:22.972363 (+ 1829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:22.972364 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:22.972365 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:22.972371 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:22.973809 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:22.974214 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:22.974215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:22.974215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:22.974278 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:22.974482 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:22.974483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:22.974483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:22.974584 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:22.974635 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:22.974816 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 03:32:22.974870 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 03:32:22.975023 22843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.968016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46485 (local address 127.0.0.1:33160)
0927 03:32:22.968048 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:22.968057 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:32:22.968096 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:32:22.968281 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:32:22.968282 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:32:22.968432 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:32:22.968477 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:22.968478 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.969377 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.969378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:22.970085 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:22.970087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.970544 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.970545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:22.970620 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:22.971004 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:32:22.971007 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:32:22.972285 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:32:22.973901 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:22.973904 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:22.973904 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:22.974128 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:22.974340 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:22.974340 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:22.974341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:22.974416 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:22.974650 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:32:22.974650 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:32:22.974697 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 03:32:22.974799 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 03:32:22.974899 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14}
I20260927 03:32:22.975399 22841 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46877 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:32:22 dist-test-slave-j0hf krb5kdc[22610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:32:22.978825 22694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.971456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42548 (local address 127.0.98.62:41473)
0927 03:32:22.971498 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:22.971500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:22.971510 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:32:22.971564 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:22.971565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:22.971581 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:22.971635 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:22.972084 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.972676 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.973513 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.973852 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.973915 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:22.976361 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:22.976362 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:22.976362 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:22.976368 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:22.977766 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:22.978129 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:22.978130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:22.978131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:22.978186 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:22.978355 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:22.978356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:22.978356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:22.978489 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:22.978637 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:22.978711 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 03:32:22.978766 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 27 03:32:22 dist-test-slave-j0hf krb5kdc[22610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 03:32:22.982707 22845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.976124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36386 (local address 127.0.98.60:46877)
0927 03:32:22.976371 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:22.976373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:22.976378 (+ 5us) server_negotiation.cc:407] Connection header received
0927 03:32:22.976393 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:22.976393 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:22.976401 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:22.976462 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:22.976700 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.977144 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.977760 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.978057 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.978122 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:22.979950 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:22.979951 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:22.979951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:22.979957 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:22.981317 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:22.982002 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:22.982003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:22.982004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:22.982060 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:22.982278 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:22.982278 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:22.982279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:22.982369 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:22.982421 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:22.982593 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 03:32:22.982654 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231}
I20260927 03:32:22.982847 22843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.976070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46877 (local address 127.0.0.1:36386)
0927 03:32:22.976140 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:22.976149 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:32:22.976184 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:32:22.976478 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:32:22.976479 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:32:22.976600 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:32:22.976622 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:22.976624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.977237 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.977238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:22.977681 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:22.977682 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.978131 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.978133 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:22.978194 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:22.978560 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:32:22.978565 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:32:22.979880 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:32:22.981720 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:22.981721 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:22.981721 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:22.981926 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:22.982138 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:22.982139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:22.982139 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:22.982202 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:22.982435 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:32:22.982436 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:32:22.982484 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 03:32:22.982575 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 03:32:22.982719 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":48}
Sep 27 03:32:22 dist-test-slave-j0hf krb5kdc[22610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 03:32:22.984306 22841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 234b8abf0e394e63bb30ab7d9c64ebd2: Bootstrap starting.
I20260927 03:32:22.985261 22841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 234b8abf0e394e63bb30ab7d9c64ebd2: Neither blocks nor log segments found. Creating new log.
I20260927 03:32:22.985551 22841 log.cc:824] T 00000000000000000000000000000000 P 234b8abf0e394e63bb30ab7d9c64ebd2: Log is configured to *not* fsync() on all Append() calls
I20260927 03:32:22.986320 22841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 234b8abf0e394e63bb30ab7d9c64ebd2: No bootstrap required, opened a new log
I20260927 03:32:22.986631 22771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:22.979632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33174 (local address 127.0.98.61:46485)
0927 03:32:22.979665 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:22.979667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:22.979705 (+ 38us) server_negotiation.cc:407] Connection header received
0927 03:32:22.979783 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:22.979784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:22.979791 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:22.979832 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:22.980302 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.980740 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.981410 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:22.981598 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:22.981670 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:22.984085 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:22.984086 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:22.984087 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:22.984093 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:22.985485 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:22.985880 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:22.985881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:22.985882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:22.985937 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:22.986143 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:22.986144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:22.986144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:22.986257 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:22.986431 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:22.986465 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 03:32:22.986537 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 03:32:22.988130 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:32:22.988503 22841 raft_consensus.cc:348] T 00000000000000000000000000000000 P 234b8abf0e394e63bb30ab7d9c64ebd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cac1ec7efb4c4d8189c618cd53daed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41473 } } peers { permanent_uuid: "3c777c75e2c147bf9ae2b115a6ccea87" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "234b8abf0e394e63bb30ab7d9c64ebd2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46877 } }
I20260927 03:32:22.988629 22841 raft_consensus.cc:729] T 00000000000000000000000000000000 P 234b8abf0e394e63bb30ab7d9c64ebd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 234b8abf0e394e63bb30ab7d9c64ebd2, State: Initialized, Role: FOLLOWER
I20260927 03:32:22.988715 22841 consensus_queue.cc:260] T 00000000000000000000000000000000 P 234b8abf0e394e63bb30ab7d9c64ebd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cac1ec7efb4c4d8189c618cd53daed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41473 } } peers { permanent_uuid: "3c777c75e2c147bf9ae2b115a6ccea87" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "234b8abf0e394e63bb30ab7d9c64ebd2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46877 } }
I20260927 03:32:22.989039 22847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 234b8abf0e394e63bb30ab7d9c64ebd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cac1ec7efb4c4d8189c618cd53daed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41473 } } peers { permanent_uuid: "3c777c75e2c147bf9ae2b115a6ccea87" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "234b8abf0e394e63bb30ab7d9c64ebd2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46877 } } }
I20260927 03:32:22.989069 22841 sys_catalog.cc:564] T 00000000000000000000000000000000 P 234b8abf0e394e63bb30ab7d9c64ebd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:32:22.989157 22847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 234b8abf0e394e63bb30ab7d9c64ebd2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:32:22.989274 22665 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "68cac1ec7efb4c4d8189c618cd53daed"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42548
W20260927 03:32:22.990993 22858 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 234b8abf0e394e63bb30ab7d9c64ebd2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:32:22.991077 22858 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:32:22.991250 22773 master_runner.cc:186] Error getting master registration for 127.0.98.62:41473: OK, instance_id { permanent_uuid: "68cac1ec7efb4c4d8189c618cd53daed" instance_seqno: 1790479942664055 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 03:32:22.993583 22773 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 03:32:23.007143 22691 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46877 } attempt: 2
Sep 27 03:32:23 dist-test-slave-j0hf krb5kdc[22610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 03:32:23.014945 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:23.007536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46877 (local address 127.0.0.1:36396)
0927 03:32:23.007605 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:23.007619 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:32:23.007669 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:32:23.007799 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:32:23.007800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:32:23.007993 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:32:23.008068 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:23.008070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:23.009045 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:23.009046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:23.009849 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:23.009851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:23.010335 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:23.010336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:23.010390 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:23.010878 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:32:23.010883 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:32:23.012185 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:32:23.013860 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:23.013861 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:23.013861 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:23.014033 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:23.014268 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:23.014269 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:23.014269 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:23.014329 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:23.014609 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:32:23.014610 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:32:23.014657 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 03:32:23.014742 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 03:32:23.014840 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 03:32:23.014989 22845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:23.007635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36396 (local address 127.0.98.60:46877)
0927 03:32:23.007693 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:23.007695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:23.007706 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:32:23.007732 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:23.007733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:23.007742 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:23.007785 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:23.008161 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:23.008945 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:23.009940 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:23.010251 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:23.010329 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:23.012279 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:23.012281 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:23.012281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:23.012288 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:23.013758 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:23.014128 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:23.014129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:23.014129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:23.014183 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:23.014380 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:23.014381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:23.014382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:23.014520 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:23.014615 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:23.014798 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 03:32:23.014892 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:32:23.016383 22691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68cac1ec7efb4c4d8189c618cd53daed: Bootstrap starting.
I20260927 03:32:23.017042 22691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68cac1ec7efb4c4d8189c618cd53daed: Neither blocks nor log segments found. Creating new log.
I20260927 03:32:23.017349 22691 log.cc:824] T 00000000000000000000000000000000 P 68cac1ec7efb4c4d8189c618cd53daed: Log is configured to *not* fsync() on all Append() calls
I20260927 03:32:23.018154 22691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68cac1ec7efb4c4d8189c618cd53daed: No bootstrap required, opened a new log
I20260927 03:32:23.019142 22691 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68cac1ec7efb4c4d8189c618cd53daed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cac1ec7efb4c4d8189c618cd53daed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41473 } } peers { permanent_uuid: "3c777c75e2c147bf9ae2b115a6ccea87" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "234b8abf0e394e63bb30ab7d9c64ebd2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46877 } }
I20260927 03:32:23.019279 22691 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68cac1ec7efb4c4d8189c618cd53daed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68cac1ec7efb4c4d8189c618cd53daed, State: Initialized, Role: FOLLOWER
I20260927 03:32:23.019367 22691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68cac1ec7efb4c4d8189c618cd53daed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cac1ec7efb4c4d8189c618cd53daed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41473 } } peers { permanent_uuid: "3c777c75e2c147bf9ae2b115a6ccea87" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "234b8abf0e394e63bb30ab7d9c64ebd2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46877 } }
I20260927 03:32:23.019644 22861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68cac1ec7efb4c4d8189c618cd53daed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cac1ec7efb4c4d8189c618cd53daed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41473 } } peers { permanent_uuid: "3c777c75e2c147bf9ae2b115a6ccea87" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "234b8abf0e394e63bb30ab7d9c64ebd2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46877 } } }
I20260927 03:32:23.019717 22691 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68cac1ec7efb4c4d8189c618cd53daed [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:32:23.019719 22861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68cac1ec7efb4c4d8189c618cd53daed [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:32:23.019860 22665 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68cac1ec7efb4c4d8189c618cd53daed [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 03:32:23.019970 22665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68cac1ec7efb4c4d8189c618cd53daed [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:32:23.020784 22665 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68cac1ec7efb4c4d8189c618cd53daed [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cac1ec7efb4c4d8189c618cd53daed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41473 } } peers { permanent_uuid: "3c777c75e2c147bf9ae2b115a6ccea87" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "234b8abf0e394e63bb30ab7d9c64ebd2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46877 } }
I20260927 03:32:23.021420 22815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68cac1ec7efb4c4d8189c618cd53daed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "234b8abf0e394e63bb30ab7d9c64ebd2"
I20260927 03:32:23.021534 22815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 234b8abf0e394e63bb30ab7d9c64ebd2 [term 0 FOLLOWER]: Advancing to term 1
W20260927 03:32:23.021859 22872 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 68cac1ec7efb4c4d8189c618cd53daed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:32:23.021905 22872 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:32:23.021181 22665 leader_election.cc:288] T 00000000000000000000000000000000 P 68cac1ec7efb4c4d8189c618cd53daed [CANDIDATE]: Term 1 election: Requested vote from peers 3c777c75e2c147bf9ae2b115a6ccea87 (127.0.98.61:46485), 234b8abf0e394e63bb30ab7d9c64ebd2 (127.0.98.60:46877)
I20260927 03:32:23.022347 22815 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 234b8abf0e394e63bb30ab7d9c64ebd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68cac1ec7efb4c4d8189c618cd53daed in term 1.
I20260927 03:32:23.023667 22640 leader_election.cc:302] T 00000000000000000000000000000000 P 68cac1ec7efb4c4d8189c618cd53daed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 234b8abf0e394e63bb30ab7d9c64ebd2, 68cac1ec7efb4c4d8189c618cd53daed; no voters:
I20260927 03:32:23.023785 22861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68cac1ec7efb4c4d8189c618cd53daed [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:32:23.023905 22861 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68cac1ec7efb4c4d8189c618cd53daed [term 1 LEADER]: Becoming Leader. State: Replica: 68cac1ec7efb4c4d8189c618cd53daed, State: Running, Role: LEADER
I20260927 03:32:23.023977 22861 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68cac1ec7efb4c4d8189c618cd53daed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cac1ec7efb4c4d8189c618cd53daed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41473 } } peers { permanent_uuid: "3c777c75e2c147bf9ae2b115a6ccea87" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "234b8abf0e394e63bb30ab7d9c64ebd2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46877 } }
I20260927 03:32:23.024377 22861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68cac1ec7efb4c4d8189c618cd53daed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68cac1ec7efb4c4d8189c618cd53daed. Latest consensus state: current_term: 1 leader_uuid: "68cac1ec7efb4c4d8189c618cd53daed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cac1ec7efb4c4d8189c618cd53daed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41473 } } peers { permanent_uuid: "3c777c75e2c147bf9ae2b115a6ccea87" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "234b8abf0e394e63bb30ab7d9c64ebd2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46877 } } }
I20260927 03:32:23.024453 22861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68cac1ec7efb4c4d8189c618cd53daed [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:23.022778 22623 master_runner.cc:186] Error getting master registration for 127.0.98.61:46485: OK, instance_id { permanent_uuid: "3c777c75e2c147bf9ae2b115a6ccea87" instance_seqno: 1790479942808796 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 03:32:23.025146 22623 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 03:32:23.031978 22880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:32:23.032166 22880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:32:23.033849 22815 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 234b8abf0e394e63bb30ab7d9c64ebd2 [term 1 FOLLOWER]: Refusing update from remote peer 68cac1ec7efb4c4d8189c618cd53daed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:32:23.034237 22878 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 68cac1ec7efb4c4d8189c618cd53daed [LEADER]: Connected to new peer: Peer: permanent_uuid: "234b8abf0e394e63bb30ab7d9c64ebd2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:32:23.035681 22847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 234b8abf0e394e63bb30ab7d9c64ebd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68cac1ec7efb4c4d8189c618cd53daed. Latest consensus state: current_term: 1 leader_uuid: "68cac1ec7efb4c4d8189c618cd53daed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cac1ec7efb4c4d8189c618cd53daed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41473 } } peers { permanent_uuid: "3c777c75e2c147bf9ae2b115a6ccea87" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "234b8abf0e394e63bb30ab7d9c64ebd2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46877 } } }
I20260927 03:32:23.035770 22847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 234b8abf0e394e63bb30ab7d9c64ebd2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:32:23.042026 22845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:23.026017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36404 (local address 127.0.98.60:46877)
0927 03:32:23.026054 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:23.026056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:23.026407 (+ 351us) server_negotiation.cc:407] Connection header received
0927 03:32:23.026495 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:23.026496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:23.026505 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:23.026533 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:23.027234 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:23.028017 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:23.029929 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:23.030222 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:23.030264 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:23.034750 (+ 4486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:23.034751 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:23.034752 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:23.034759 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:23.036647 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:23.037173 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:23.037174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:23.037175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:23.037228 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:23.040166 (+ 2938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:23.040168 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:23.040168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:23.040309 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:23.040385 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:23.041857 (+ 1472us) server_negotiation.cc:299] Negotiation successful
0927 03:32:23.041933 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:32:23.043682 22694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:23.025729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42562 (local address 127.0.98.62:41473)
0927 03:32:23.025768 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:23.025770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:23.026338 (+ 568us) server_negotiation.cc:407] Connection header received
0927 03:32:23.026496 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:23.026496 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:23.026517 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:23.026546 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:23.027693 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:23.028383 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:23.029874 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:23.030179 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:23.030222 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:23.037155 (+ 6933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:23.037157 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:23.037157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:23.037164 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:23.039342 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:23.040440 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:23.040441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:23.040441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:23.040498 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:23.042364 (+ 1866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:23.042365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:23.042366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:23.042629 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:23.042791 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:23.043459 (+ 668us) server_negotiation.cc:299] Negotiation successful
0927 03:32:23.043568 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:32:23.044600 22884 mvcc.cc:205] Tried to move back new op lower bound from 7333805846663946240 to 7333805846626922496. Current Snapshot: MvccSnapshot[applied={T|T < 7333805846663946240}]
I20260927 03:32:23.044749 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68cac1ec7efb4c4d8189c618cd53daed [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "68cac1ec7efb4c4d8189c618cd53daed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cac1ec7efb4c4d8189c618cd53daed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41473 } } peers { permanent_uuid: "3c777c75e2c147bf9ae2b115a6ccea87" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "234b8abf0e394e63bb30ab7d9c64ebd2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46877 } } }
I20260927 03:32:23.044828 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68cac1ec7efb4c4d8189c618cd53daed [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:23.045307 22847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 234b8abf0e394e63bb30ab7d9c64ebd2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "68cac1ec7efb4c4d8189c618cd53daed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cac1ec7efb4c4d8189c618cd53daed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41473 } } peers { permanent_uuid: "3c777c75e2c147bf9ae2b115a6ccea87" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "234b8abf0e394e63bb30ab7d9c64ebd2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46877 } } }
I20260927 03:32:23.045411 22847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 234b8abf0e394e63bb30ab7d9c64ebd2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:32:23.045436 22885 mvcc.cc:205] Tried to move back new op lower bound from 7333805846663946240 to 7333805846626922496. Current Snapshot: MvccSnapshot[applied={T|T < 7333805846663946240}]
I20260927 03:32:23.045488 22766 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46877 } attempt: 3
I20260927 03:32:23.046236 22880 catalog_manager.cc:1347] Generated new cluster ID: dc81bf473faf4304847fdf821f8c026c
I20260927 03:32:23.046280 22880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:32:23 dist-test-slave-j0hf krb5kdc[22610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 03:32:23.054157 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:23.045713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46877 (local address 127.0.0.1:36406)
0927 03:32:23.045755 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:23.045764 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:32:23.045797 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:32:23.046024 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:32:23.046026 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:32:23.046175 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:32:23.046216 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:23.046218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:23.047159 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:23.047161 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:23.047958 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:23.047960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:23.049370 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:23.049371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:23.049426 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:23.049843 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:32:23.049854 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:32:23.051312 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:32:23.053128 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:23.053128 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:23.053129 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:23.053320 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:23.053514 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:23.053515 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:23.053516 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:23.053579 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:23.053853 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:32:23.053853 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:32:23.053891 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 03:32:23.053948 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 03:32:23.054057 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 03:32:23.054692 22845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:23.045785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36406 (local address 127.0.98.60:46877)
0927 03:32:23.045889 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:23.045891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:23.045899 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:32:23.045917 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:23.045918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:23.045927 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:23.045981 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:23.046340 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:23.047069 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:23.048942 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:23.049252 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:23.049311 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:23.051405 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:23.051407 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:23.051407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:23.051413 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:23.053043 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:23.053396 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:23.053397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:23.053397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:23.053447 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:23.053653 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:23.053653 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:23.053654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:23.053770 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:23.053831 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:23.054526 (+ 695us) server_negotiation.cc:299] Negotiation successful
0927 03:32:23.054608 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 03:32:23.056412 22766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c777c75e2c147bf9ae2b115a6ccea87: Bootstrap starting.
I20260927 03:32:23.057163 22766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c777c75e2c147bf9ae2b115a6ccea87: Neither blocks nor log segments found. Creating new log.
I20260927 03:32:23.057438 22766 log.cc:824] T 00000000000000000000000000000000 P 3c777c75e2c147bf9ae2b115a6ccea87: Log is configured to *not* fsync() on all Append() calls
I20260927 03:32:23.058063 22880 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:32:23.059013 22771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:23.025527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33180 (local address 127.0.98.61:46485)
0927 03:32:23.025570 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:23.025572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:23.026318 (+ 746us) server_negotiation.cc:407] Connection header received
0927 03:32:23.029113 (+ 2795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:23.029114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:23.029125 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:23.029154 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:23.032128 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:23.032852 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:23.037674 (+ 4822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:23.038004 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:23.038048 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:23.044177 (+ 6129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:23.044178 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:23.044179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:23.044186 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:23.048850 (+ 4664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:23.058118 (+ 9268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:23.058120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:23.058120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:23.058174 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:23.058423 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:23.058424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:23.058424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:23.058558 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:23.058635 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:23.058848 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 03:32:23.058926 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:32:23.059338 22880 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:32:23.064162 22766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c777c75e2c147bf9ae2b115a6ccea87: No bootstrap required, opened a new log
I20260927 03:32:23.066282 22766 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c777c75e2c147bf9ae2b115a6ccea87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cac1ec7efb4c4d8189c618cd53daed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41473 } } peers { permanent_uuid: "3c777c75e2c147bf9ae2b115a6ccea87" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "234b8abf0e394e63bb30ab7d9c64ebd2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46877 } }
I20260927 03:32:23.066416 22766 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c777c75e2c147bf9ae2b115a6ccea87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c777c75e2c147bf9ae2b115a6ccea87, State: Initialized, Role: FOLLOWER
I20260927 03:32:23.066493 22766 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c777c75e2c147bf9ae2b115a6ccea87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cac1ec7efb4c4d8189c618cd53daed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41473 } } peers { permanent_uuid: "3c777c75e2c147bf9ae2b115a6ccea87" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "234b8abf0e394e63bb30ab7d9c64ebd2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46877 } }
I20260927 03:32:23.066841 22766 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c777c75e2c147bf9ae2b115a6ccea87 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:32:23.067189 22740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68cac1ec7efb4c4d8189c618cd53daed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3c777c75e2c147bf9ae2b115a6ccea87"
I20260927 03:32:23.067291 22740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c777c75e2c147bf9ae2b115a6ccea87 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:32:23.068248 22740 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3c777c75e2c147bf9ae2b115a6ccea87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68cac1ec7efb4c4d8189c618cd53daed in term 1.
I20260927 03:32:23.069432 22739 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c777c75e2c147bf9ae2b115a6ccea87 [term 1 FOLLOWER]: Refusing update from remote peer 68cac1ec7efb4c4d8189c618cd53daed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:32:23.069689 22861 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 68cac1ec7efb4c4d8189c618cd53daed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c777c75e2c147bf9ae2b115a6ccea87" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:32:23.071532 22891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c777c75e2c147bf9ae2b115a6ccea87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68cac1ec7efb4c4d8189c618cd53daed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cac1ec7efb4c4d8189c618cd53daed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41473 } } peers { permanent_uuid: "3c777c75e2c147bf9ae2b115a6ccea87" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "234b8abf0e394e63bb30ab7d9c64ebd2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46877 } } }
I20260927 03:32:23.071686 22891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c777c75e2c147bf9ae2b115a6ccea87 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:32:23.071758 22891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c777c75e2c147bf9ae2b115a6ccea87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68cac1ec7efb4c4d8189c618cd53daed. Latest consensus state: current_term: 1 leader_uuid: "68cac1ec7efb4c4d8189c618cd53daed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cac1ec7efb4c4d8189c618cd53daed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41473 } } peers { permanent_uuid: "3c777c75e2c147bf9ae2b115a6ccea87" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "234b8abf0e394e63bb30ab7d9c64ebd2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46877 } } }
I20260927 03:32:23.071848 22891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c777c75e2c147bf9ae2b115a6ccea87 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:32:23.071889 22891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c777c75e2c147bf9ae2b115a6ccea87 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "68cac1ec7efb4c4d8189c618cd53daed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cac1ec7efb4c4d8189c618cd53daed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41473 } } peers { permanent_uuid: "3c777c75e2c147bf9ae2b115a6ccea87" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "234b8abf0e394e63bb30ab7d9c64ebd2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46877 } } }
I20260927 03:32:23.071950 22891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c777c75e2c147bf9ae2b115a6ccea87 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:32:23.072842 22880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68cac1ec7efb4c4d8189c618cd53daed: Generated new TSK 0
I20260927 03:32:23.072993 22880 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:32:23.077911 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68cac1ec7efb4c4d8189c618cd53daed [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "68cac1ec7efb4c4d8189c618cd53daed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cac1ec7efb4c4d8189c618cd53daed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41473 } } peers { permanent_uuid: "3c777c75e2c147bf9ae2b115a6ccea87" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46485 } } peers { permanent_uuid: "234b8abf0e394e63bb30ab7d9c64ebd2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46877 } } }
I20260927 03:32:23.078009 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68cac1ec7efb4c4d8189c618cd53daed [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:23.079259 22893 mvcc.cc:205] Tried to move back new op lower bound from 7333805846757384192 to 7333805846626922496. Current Snapshot: MvccSnapshot[applied={T|T < 7333805846663946240}]
W20260927 03:32:23.080264 22906 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c777c75e2c147bf9ae2b115a6ccea87: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:32:23.080379 22906 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260927 03:32:23.098459 22694 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:32:23.096866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42572 (local address 127.0.98.62:41473)
0927 03:32:23.096912 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:23.096914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:23.097699 (+ 785us) server_negotiation.cc:407] Connection header received
0927 03:32:23.097809 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:23.097810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:23.097839 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:23.097871 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:23.098388 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42572: BlockingRecv error: recv got EOF from 127.0.0.1:42572 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 03:32:23.099180 22845 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:32:23.097244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36410 (local address 127.0.98.60:46877)
0927 03:32:23.097285 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:23.097287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:23.098686 (+ 1399us) server_negotiation.cc:407] Connection header received
0927 03:32:23.098781 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:23.098782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:23.098793 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:23.098821 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:23.099122 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36410: BlockingRecv error: recv got EOF from 127.0.0.1:36410 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 03:32:23.099898 22771 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:32:23.097627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33188 (local address 127.0.98.61:46485)
0927 03:32:23.097673 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:23.097675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:23.099392 (+ 1717us) server_negotiation.cc:407] Connection header received
0927 03:32:23.099510 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:23.099511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:23.099522 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:23.099550 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:23.099829 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33188: BlockingRecv error: recv got EOF from 127.0.0.1:33188 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 03:32:23.101699 22771 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:32:23.101032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33194 (local address 127.0.98.61:46485)
0927 03:32:23.101067 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:23.101069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:23.101237 (+ 168us) server_negotiation.cc:407] Connection header received
0927 03:32:23.101334 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:23.101335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:23.101352 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:23.101382 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:23.101659 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33194: BlockingRecv error: recv got EOF from 127.0.0.1:33194 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 03:32:23.104660 22771 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:32:23.103930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33208 (local address 127.0.98.61:46485)
0927 03:32:23.103960 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:23.103961 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:23.104197 (+ 236us) server_negotiation.cc:407] Connection header received
0927 03:32:23.104289 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:23.104290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:23.104299 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:23.104326 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:23.104619 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33208: BlockingRecv error: recv got EOF from 127.0.0.1:33208 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260927 03:32:23.108467 22771 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:32:23.107761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33214 (local address 127.0.98.61:46485)
0927 03:32:23.107793 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:23.107795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:23.107986 (+ 191us) server_negotiation.cc:407] Connection header received
0927 03:32:23.108090 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:23.108091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:23.108101 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:23.108130 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:23.108424 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33214: BlockingRecv error: recv got EOF from 127.0.0.1:33214 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 03:32:23.114274 22771 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:32:23.113550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33222 (local address 127.0.98.61:46485)
0927 03:32:23.113586 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:23.113587 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:23.113801 (+ 214us) server_negotiation.cc:407] Connection header received
0927 03:32:23.113888 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:23.113888 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:23.113898 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:23.113924 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:23.114229 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33222: BlockingRecv error: recv got EOF from 127.0.0.1:33222 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 03:32:23.124382 22771 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:32:23.123537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33236 (local address 127.0.98.61:46485)
0927 03:32:23.123576 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:23.123578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:23.123821 (+ 243us) server_negotiation.cc:407] Connection header received
0927 03:32:23.123920 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:23.123921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:23.123932 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:23.123959 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:23.124330 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33236: BlockingRecv error: recv got EOF from 127.0.0.1:33236 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 03:32:23.143122 22771 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:32:23.141974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33244 (local address 127.0.98.61:46485)
0927 03:32:23.142019 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:23.142021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:23.142402 (+ 381us) server_negotiation.cc:407] Connection header received
0927 03:32:23.142521 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:23.142522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:23.142536 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:23.142570 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:23.143025 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33244: BlockingRecv error: recv got EOF from 127.0.0.1:33244 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 03:32:23.177721 22771 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:32:23.176694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33254 (local address 127.0.98.61:46485)
0927 03:32:23.176744 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:23.176746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:23.177046 (+ 300us) server_negotiation.cc:407] Connection header received
0927 03:32:23.177173 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:23.177174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:23.177188 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:23.177224 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:23.177656 (+ 432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33254: BlockingRecv error: recv got EOF from 127.0.0.1:33254 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 03:32:23.243964 22771 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:32:23.243098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33262 (local address 127.0.98.61:46485)
0927 03:32:23.243169 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:23.243171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:23.243303 (+ 132us) server_negotiation.cc:407] Connection header received
0927 03:32:23.243377 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:23.243378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:23.243389 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:23.243483 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:23.243904 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33262: BlockingRecv error: recv got EOF from 127.0.0.1:33262 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 03:32:23.373934 22771 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:32:23.373108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33278 (local address 127.0.98.61:46485)
0927 03:32:23.373195 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:23.373197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:23.373305 (+ 108us) server_negotiation.cc:407] Connection header received
0927 03:32:23.373388 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:23.373389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:23.373403 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:23.373470 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:23.373867 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33278: BlockingRecv error: recv got EOF from 127.0.0.1:33278 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 03:32:23.631963 22771 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:32:23.631238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33290 (local address 127.0.98.61:46485)
0927 03:32:23.631307 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:23.631309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:23.631319 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:32:23.631372 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:23.631373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:23.631391 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:23.631522 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:23.631896 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33290: BlockingRecv error: recv got EOF from 127.0.0.1:33290 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 03:32:23.991564 22858 catalog_manager.cc:1259] Loaded cluster ID: dc81bf473faf4304847fdf821f8c026c
I20260927 03:32:23.991632 22858 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 234b8abf0e394e63bb30ab7d9c64ebd2: loading cluster ID for follower catalog manager: success
I20260927 03:32:23.992400 22858 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 234b8abf0e394e63bb30ab7d9c64ebd2: acquiring CA information for follower catalog manager: success
I20260927 03:32:23.992693 22858 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 234b8abf0e394e63bb30ab7d9c64ebd2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 03:32:24.080895 22906 catalog_manager.cc:1259] Loaded cluster ID: dc81bf473faf4304847fdf821f8c026c
I20260927 03:32:24.080977 22906 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3c777c75e2c147bf9ae2b115a6ccea87: loading cluster ID for follower catalog manager: success
I20260927 03:32:24.081825 22906 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3c777c75e2c147bf9ae2b115a6ccea87: acquiring CA information for follower catalog manager: success
I20260927 03:32:24.082151 22906 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3c777c75e2c147bf9ae2b115a6ccea87: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 03:32:24.147814 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:24.145199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33298 (local address 127.0.98.61:46485)
0927 03:32:24.145314 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:24.145316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:24.145364 (+ 48us) server_negotiation.cc:407] Connection header received
0927 03:32:24.145426 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:24.145427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:24.145478 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:24.145544 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:32:24.145693 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:24.146310 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:24.147057 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:24.147248 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:24.147297 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:24.147452 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:32:24.147561 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:32:24.147620 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:24.147662 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 03:32:24.147680 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":58,"threads_started":1}
I20260927 03:32:24.152251 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:24.149803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36420 (local address 127.0.98.60:46877)
0927 03:32:24.149911 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:24.149913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:24.149922 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:32:24.149966 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:24.149967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:24.150004 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:24.150077 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:32:24.150241 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:24.150714 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:24.151516 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:24.151701 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:24.151753 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:24.151888 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:32:24.151993 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:32:24.152050 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:24.152105 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 03:32:24.152121 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":67,"threads_started":1}
I20260927 03:32:24.156810 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:24.154166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36428 (local address 127.0.98.60:46877)
0927 03:32:24.154228 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:24.154231 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:24.154239 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:32:24.154259 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:24.154260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:24.154271 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:24.154329 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:32:24.154485 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:24.155210 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:24.156039 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:24.156315 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:24.156380 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:24.156529 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:32:24.156563 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:32:24.156639 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:24.156704 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 03:32:24.156732 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:32:24.157012 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:24.153939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33300 (local address 127.0.98.61:46485)
0927 03:32:24.154031 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:24.154032 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:24.154070 (+ 38us) server_negotiation.cc:407] Connection header received
0927 03:32:24.154139 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:24.154140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:24.154151 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:24.154224 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:32:24.154435 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:24.155209 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:24.156219 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:24.156506 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:24.156571 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:24.156727 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:32:24.156761 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:32:24.156793 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:24.156906 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 03:32:24.156936 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 03:32:24.157287 22988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:24.153843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42584 (local address 127.0.98.62:41473)
0927 03:32:24.153947 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:24.153949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:24.153960 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:32:24.154185 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:24.154187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:24.154249 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:24.154352 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:32:24.154523 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:24.155029 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:24.156147 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:24.156461 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:24.156537 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:24.156799 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:32:24.156921 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:32:24.156982 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:24.157153 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 03:32:24.157180 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":48,"threads_started":1}
I20260927 03:32:24.159068 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 22623
I20260927 03:32:24.164852 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 22698
I20260927 03:32:24.169791 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 22773
2026-09-27T03:32:24Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2796 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:32:24 dist-test-slave-j0hf krb5kdc[22992](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:32:24 dist-test-slave-j0hf krb5kdc[22992](info): set up 2 sockets
Sep 27 03:32:24 dist-test-slave-j0hf krb5kdc[22992](info): commencing operation
krb5kdc: starting...
W20260927 03:32:25.211800 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/krb5kdc/test-user.keytab.
Sep 27 03:32:25 dist-test-slave-j0hf krb5kdc[22992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:32:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:32:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 03:32:25.331176 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42863
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42863,127.0.98.61:46741,127.0.98.60:43467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:32:25.414644 23005 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 03:32:25.414945 23005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:32:25.414968 23005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:32:25.414990 23005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:32:25.416256 23005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:32:25.416306 23005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:32:25.416322 23005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:32:25.416337 23005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:32:25.416352 23005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:32:25.417893 23005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42863,127.0.98.61:46741,127.0.98.60:43467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:32:25.418169 23005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:32:25.418426 23005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:32:25.423256 23013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:32:25.423352 23005 server_base.cc:1034] running on GCE node
W20260927 03:32:25.423352 23010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:25.423494 23011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:32:25.423760 23005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:32:25.424044 23005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:32:25.425223 23005 hybrid_clock.cc:648] HybridClock initialized: now 1790479945425200 us; error 41 us; skew 500 ppm
Sep 27 03:32:25 dist-test-slave-j0hf krb5kdc[22992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:32:25.427686 23005 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 03:32:25.428108 23005 webserver.cc:460] Webserver started at http://127.0.98.62:38001/ using document root <none> and password file <none>
I20260927 03:32:25.428341 23005 fs_manager.cc:359] Metadata directory not provided
I20260927 03:32:25.428409 23005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:32:25.428561 23005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:32:25.429626 23005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-0/data/instance:
uuid: "28951365811142f8ac5b001938e52de0"
format_stamp: "Formatted at 2026-09-27 03:32:25 on dist-test-slave-j0hf"
server_key: "1f0e944b80fe0a9633b690fdfc97fbb8"
server_key_iv: "2185c71755f4fa4aa5e6a91770d4b634"
server_key_version: "encryptionkey@0"
I20260927 03:32:25.429994 23005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-0/wal/instance:
uuid: "28951365811142f8ac5b001938e52de0"
format_stamp: "Formatted at 2026-09-27 03:32:25 on dist-test-slave-j0hf"
server_key: "1f0e944b80fe0a9633b690fdfc97fbb8"
server_key_iv: "2185c71755f4fa4aa5e6a91770d4b634"
server_key_version: "encryptionkey@0"
I20260927 03:32:25.431295 23005 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:32:25.432046 23020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:32:25.432207 23005 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:32:25.432304 23005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-0/wal
uuid: "28951365811142f8ac5b001938e52de0"
format_stamp: "Formatted at 2026-09-27 03:32:25 on dist-test-slave-j0hf"
server_key: "1f0e944b80fe0a9633b690fdfc97fbb8"
server_key_iv: "2185c71755f4fa4aa5e6a91770d4b634"
server_key_version: "encryptionkey@0"
I20260927 03:32:25.432386 23005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:32:25.448791 23005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:32:25.453581 23005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:32:25.453763 23005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:32:25.457739 23005 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42863
I20260927 03:32:25.457777 23072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42863 every 8 connection(s)
I20260927 03:32:25.458129 23005 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-0/data/info.pb
I20260927 03:32:25.459131 23073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:32:25.460780 23073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:32:25.465844 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 23005
I20260927 03:32:25.465937 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-0/wal/instance
I20260927 03:32:25.466148 392 external_mini_cluster.cc:1428] Setting key 3524be61aad420bc199cbad7d6bdd192
Sep 27 03:32:25 dist-test-slave-j0hf krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:32:25.470260 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.461640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38758 (local address 127.0.98.62:42863)
0927 03:32:25.461756 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:25.461761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:25.461796 (+ 35us) server_negotiation.cc:407] Connection header received
0927 03:32:25.462252 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:25.462254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:25.462270 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:25.462339 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:25.462857 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.463773 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.464574 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.464864 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.464942 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:25.467019 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:25.467025 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:25.467026 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:25.467033 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:25.468643 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:25.469093 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:25.469095 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:25.469095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:25.469177 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:25.469424 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:25.469424 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:25.469425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:25.469668 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:25.469738 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:25.469981 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 03:32:25.470096 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":88,"thread_start_us":62,"threads_started":1}
I20260927 03:32:25.470361 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.461473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42863 (local address 127.0.0.1:38758)
0927 03:32:25.461643 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:25.461681 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 03:32:25.462100 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:32:25.462365 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:32:25.462367 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:32:25.462588 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:32:25.462752 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:25.462755 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.463851 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.463852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:25.464505 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:25.464506 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.464950 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.464951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:25.465001 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:25.465456 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:32:25.465463 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:32:25.466942 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:32:25.468725 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:25.468732 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:25.468733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:25.469012 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:25.469243 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:25.469244 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:25.469245 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:25.469358 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:25.469758 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:32:25.469766 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:32:25.469847 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 03:32:25.469955 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 03:32:25.470185 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":69,"thread_start_us":82,"threads_started":1}
I20260927 03:32:25.471057 23073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46741 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:32:25.471407 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.471297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46741 (local address 127.0.0.1:52878)
0927 03:32:25.471344 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:25.471384 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 03:32:25.471618 23021 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46741: connect: Connection refused (error 111)
I20260927 03:32:25.472028 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.471945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46741 (local address 127.0.0.1:52886)
0927 03:32:25.471988 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:25.472012 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 03:32:25.472168 23073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46741: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46741: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 03:32:25.491101 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46741
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42863,127.0.98.61:46741,127.0.98.60:43467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:32:25.521410 23073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46741 } attempt: 1
I20260927 03:32:25.521899 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.521768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46741 (local address 127.0.0.1:52896)
0927 03:32:25.521829 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:25.521871 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 03:32:25.522240 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.522149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46741 (local address 127.0.0.1:52910)
0927 03:32:25.522194 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:25.522217 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 03:32:25.522322 23073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46741: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46741: connect: Connection refused (error 111)
W20260927 03:32:25.570883 23080 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 03:32:25.571091 23080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:32:25.571108 23080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:32:25.571130 23080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:32:25.572307 23080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:32:25.572350 23080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:32:25.572363 23080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:32:25.572409 23080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:32:25.572439 23080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:32:25.573946 23080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42863,127.0.98.61:46741,127.0.98.60:43467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:32:25.574203 23080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:32:25.574430 23080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:32:25.579068 23085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:25.579118 23088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:25.579118 23086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:32:25.579454 23080 server_base.cc:1034] running on GCE node
I20260927 03:32:25.579704 23080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:32:25.579979 23080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:32:25.581118 23080 hybrid_clock.cc:648] HybridClock initialized: now 1790479945581106 us; error 28 us; skew 500 ppm
Sep 27 03:32:25 dist-test-slave-j0hf krb5kdc[22992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:32:25.583665 23080 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 03:32:25.584128 23080 webserver.cc:460] Webserver started at http://127.0.98.61:42771/ using document root <none> and password file <none>
I20260927 03:32:25.584353 23080 fs_manager.cc:359] Metadata directory not provided
I20260927 03:32:25.584419 23080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:32:25.584571 23080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:32:25.585572 23080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-1/data/instance:
uuid: "c3462c6548024245a05897392fad6967"
format_stamp: "Formatted at 2026-09-27 03:32:25 on dist-test-slave-j0hf"
server_key: "2b2216fc82ad2cf2c505b62057dc6927"
server_key_iv: "2701d4da6088b0f5899ca4931c305db1"
server_key_version: "encryptionkey@0"
I20260927 03:32:25.585933 23080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-1/wal/instance:
uuid: "c3462c6548024245a05897392fad6967"
format_stamp: "Formatted at 2026-09-27 03:32:25 on dist-test-slave-j0hf"
server_key: "2b2216fc82ad2cf2c505b62057dc6927"
server_key_iv: "2701d4da6088b0f5899ca4931c305db1"
server_key_version: "encryptionkey@0"
I20260927 03:32:25.587285 23080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:32:25.588100 23095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:32:25.588285 23080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:32:25.588382 23080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-1/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-1/wal
uuid: "c3462c6548024245a05897392fad6967"
format_stamp: "Formatted at 2026-09-27 03:32:25 on dist-test-slave-j0hf"
server_key: "2b2216fc82ad2cf2c505b62057dc6927"
server_key_iv: "2701d4da6088b0f5899ca4931c305db1"
server_key_version: "encryptionkey@0"
I20260927 03:32:25.588480 23080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:32:25.590449 23073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46741 } attempt: 2
I20260927 03:32:25.590808 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.590689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46741 (local address 127.0.0.1:52920)
0927 03:32:25.590745 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:25.590780 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 03:32:25.591115 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.591003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46741 (local address 127.0.0.1:52926)
0927 03:32:25.591078 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:25.591100 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 03:32:25.591198 23073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46741: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46741: connect: Connection refused (error 111)
I20260927 03:32:25.607151 23080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:32:25.609524 23080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:32:25.609699 23080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:32:25.613837 23080 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46741
I20260927 03:32:25.613888 23147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46741 every 8 connection(s)
I20260927 03:32:25.614259 23080 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-1/data/info.pb
I20260927 03:32:25.615195 23148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:32:25.615756 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 23080
I20260927 03:32:25.615865 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-1/wal/instance
I20260927 03:32:25.616073 392 external_mini_cluster.cc:1428] Setting key 01083cd6a88706d8ef2f9c0a7df6430d
I20260927 03:32:25.617553 23148 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:32:25 dist-test-slave-j0hf krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260927 03:32:25.627105 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.618364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38760 (local address 127.0.98.62:42863)
0927 03:32:25.618435 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:25.618437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:25.618696 (+ 259us) server_negotiation.cc:407] Connection header received
0927 03:32:25.619179 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:25.619180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:25.619190 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:25.619274 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:25.620075 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.620652 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.621628 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.621850 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.621937 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:25.624452 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:25.624454 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:25.624454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:25.624460 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:25.625765 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:25.626124 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:25.626124 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:25.626125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:25.626180 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:25.626462 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:25.626463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:25.626463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:25.626563 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:25.626614 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:25.626956 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 03:32:25.627043 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:32:25.627332 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.618337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42863 (local address 127.0.0.1:38760)
0927 03:32:25.618545 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:25.618595 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 03:32:25.619078 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:32:25.619361 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:32:25.619365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:32:25.619720 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:32:25.619969 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:25.619974 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.620744 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.620745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:25.621537 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:25.621539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.621955 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.621957 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:25.622097 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:25.622660 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:32:25.622669 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:32:25.624359 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:32:25.625839 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:25.625845 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:25.625847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:25.626056 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:25.626260 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:25.626261 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:25.626261 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:25.626401 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:25.626620 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:32:25.626625 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:32:25.626792 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 03:32:25.626908 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 03:32:25.627097 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":101,"threads_started":1}
I20260927 03:32:25.627940 23148 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46741 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:32:25 dist-test-slave-j0hf krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 03:32:25.635718 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.628207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46741 (local address 127.0.0.1:52932)
0927 03:32:25.628248 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:25.628257 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:32:25.628289 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:32:25.628591 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:32:25.628593 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:32:25.628743 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:32:25.628808 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:25.628810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.629729 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.629729 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:25.630197 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:25.630198 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.630898 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.630899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:25.630951 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:25.631377 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:32:25.631381 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:32:25.632815 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:32:25.634491 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:25.634492 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:25.634492 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:25.634722 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:25.635019 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:25.635021 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:25.635022 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:25.635079 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:25.635373 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:32:25.635373 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:32:25.635416 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 03:32:25.635527 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 03:32:25.635620 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 03:32:25.635815 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.628266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52932 (local address 127.0.98.61:46741)
0927 03:32:25.628424 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:25.628429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:25.628449 (+ 20us) server_negotiation.cc:407] Connection header received
0927 03:32:25.628473 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:25.628474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:25.628489 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:25.628566 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:25.628876 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.629638 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.630276 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.630820 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.630897 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:25.632907 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:25.632913 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:25.632914 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:25.632919 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:25.634406 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:25.634836 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:25.634838 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:25.634838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:25.634939 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:25.635158 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:25.635159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:25.635160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:25.635306 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:25.635372 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:25.635599 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 03:32:25.635671 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
I20260927 03:32:25.636407 23148 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43467 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:32:25.636761 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.636633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43467 (local address 127.0.0.1:55780)
0927 03:32:25.636692 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:25.636732 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 03:32:25.636917 23096 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43467: connect: Connection refused (error 111)
I20260927 03:32:25.637347 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.637251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43467 (local address 127.0.0.1:55794)
0927 03:32:25.637300 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:25.637328 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 03:32:25.637465 23148 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43467: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43467: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 03:32:25.642304 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7YYkzb/build/release/bin/kudu
/tmp/dist-test-task7YYkzb/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43467
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42863,127.0.98.61:46741,127.0.98.60:43467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:32:25.682344 23073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46741 } attempt: 3
Sep 27 03:32:25 dist-test-slave-j0hf krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 03:32:25.686695 23148 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43467 } attempt: 1
I20260927 03:32:25.687093 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.686967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43467 (local address 127.0.0.1:55800)
0927 03:32:25.687027 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:25.687062 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 03:32:25.687793 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.687490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43467 (local address 127.0.0.1:55816)
0927 03:32:25.687742 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:25.687768 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235}
W20260927 03:32:25.687935 23148 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43467: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43467: connect: Connection refused (error 111)
I20260927 03:32:25.689901 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.682748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46741 (local address 127.0.0.1:52936)
0927 03:32:25.682818 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:25.682832 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:32:25.682911 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:32:25.683084 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:32:25.683085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:32:25.683249 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:32:25.683315 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:25.683317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.684019 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.684020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:25.684546 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:25.684548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.684944 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.684945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:25.685026 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:25.685569 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:32:25.685573 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:32:25.687088 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:32:25.688934 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:25.688935 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:25.688935 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:25.689103 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:25.689302 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:25.689303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:25.689303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:25.689354 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:25.689607 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:32:25.689608 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:32:25.689648 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 03:32:25.689700 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 03:32:25.689779 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 03:32:25.689914 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.682833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52936 (local address 127.0.98.61:46741)
0927 03:32:25.682910 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:25.682912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:25.682925 (+ 13us) server_negotiation.cc:407] Connection header received
0927 03:32:25.682983 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:25.682984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:25.682998 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:25.683064 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:25.683404 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.683931 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.684636 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.684854 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.684943 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:25.687239 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:25.687241 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:25.687241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:25.687250 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:25.688852 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:25.689175 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:25.689176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:25.689177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:25.689238 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:25.689412 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:25.689413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:25.689413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:25.689548 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:25.689613 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:25.689749 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 03:32:25.689830 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 03:32:25.690335 23073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43467 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:32:25.690644 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.690563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43467 (local address 127.0.0.1:55828)
0927 03:32:25.690594 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:25.690622 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260927 03:32:25.690920 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.690867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43467 (local address 127.0.0.1:55830)
0927 03:32:25.690888 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:25.690908 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260927 03:32:25.691035 23073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43467: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43467: connect: Connection refused (error 111)
I20260927 03:32:25.722234 23073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43467 } attempt: 1
I20260927 03:32:25.722764 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.722620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43467 (local address 127.0.0.1:55842)
0927 03:32:25.722692 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:25.722734 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 03:32:25.723169 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.723039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43467 (local address 127.0.0.1:55844)
0927 03:32:25.723104 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:25.723144 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 03:32:25.723295 23073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43467: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43467: connect: Connection refused (error 111)
W20260927 03:32:25.725276 23155 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 03:32:25.725474 23155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:32:25.725492 23155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:32:25.725512 23155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:32:25.726572 23155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:32:25.726605 23155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:32:25.726620 23155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:32:25.726634 23155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:32:25.726645 23155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:32:25.728127 23155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42863,127.0.98.61:46741,127.0.98.60:43467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:25:57 UTC on ccycloud.quasar-uvpsqz.root.comops.site
build id 4675615
I20260927 03:32:25.728338 23155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:32:25.728502 23155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:32:25.733204 23155 server_base.cc:1034] running on GCE node
W20260927 03:32:25.733188 23163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:25.733183 23160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:32:25.733188 23161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:32:25.733548 23155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:32:25.733798 23155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:32:25.734947 23155 hybrid_clock.cc:648] HybridClock initialized: now 1790479945734929 us; error 33 us; skew 500 ppm
Sep 27 03:32:25 dist-test-slave-j0hf krb5kdc[22992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:32:25.737358 23155 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 03:32:25.737831 23155 webserver.cc:460] Webserver started at http://127.0.98.60:36717/ using document root <none> and password file <none>
I20260927 03:32:25.738063 23155 fs_manager.cc:359] Metadata directory not provided
I20260927 03:32:25.738130 23155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:32:25.738268 23155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:32:25.739267 23155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-2/data/instance:
uuid: "7f9e785f54a940868a1d6a88f626a064"
format_stamp: "Formatted at 2026-09-27 03:32:25 on dist-test-slave-j0hf"
server_key: "987a2131194cd6774db6503592e003c5"
server_key_iv: "321da9ba58f5aede3f9f1b3dec2c3fa5"
server_key_version: "encryptionkey@0"
I20260927 03:32:25.739651 23155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-2/wal/instance:
uuid: "7f9e785f54a940868a1d6a88f626a064"
format_stamp: "Formatted at 2026-09-27 03:32:25 on dist-test-slave-j0hf"
server_key: "987a2131194cd6774db6503592e003c5"
server_key_iv: "321da9ba58f5aede3f9f1b3dec2c3fa5"
server_key_version: "encryptionkey@0"
I20260927 03:32:25.741000 23155 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:32:25.741770 23170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:32:25.741958 23155 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:32:25.742112 23155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-2/data,/tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-2/wal
uuid: "7f9e785f54a940868a1d6a88f626a064"
format_stamp: "Formatted at 2026-09-27 03:32:25 on dist-test-slave-j0hf"
server_key: "987a2131194cd6774db6503592e003c5"
server_key_iv: "321da9ba58f5aede3f9f1b3dec2c3fa5"
server_key_version: "encryptionkey@0"
I20260927 03:32:25.742185 23155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:32:25.750161 23155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:32:25.752391 23155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:32:25.752539 23155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:32:25.756102 23148 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43467 } attempt: 2
I20260927 03:32:25.756505 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.756391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43467 (local address 127.0.0.1:55846)
0927 03:32:25.756444 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:25.756479 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 03:32:25.756781 23155 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43467
I20260927 03:32:25.756990 23222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43467 every 8 connection(s)
I20260927 03:32:25.757205 23155 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-2/data/info.pb
I20260927 03:32:25.757928 23223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:32:25.760170 23223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:32:25 dist-test-slave-j0hf krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 03:32:25 dist-test-slave-j0hf krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:32:25.766956 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7YYkzb/build/release/bin/kudu as pid 23155
I20260927 03:32:25.767084 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7YYkzb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790479675865103-392-0/minicluster-data/master-2/wal/instance
I20260927 03:32:25.767181 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.756818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43467 (local address 127.0.0.1:55852)
0927 03:32:25.756870 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:25.756881 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:32:25.756934 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:32:25.758755 (+ 1821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:32:25.758757 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:32:25.758885 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:32:25.758931 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:25.758940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.760596 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.760597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:25.761318 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:25.761321 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.762031 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.762032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:25.762092 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:25.762580 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:32:25.762585 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:32:25.763793 (+ 1208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:32:25.765756 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:25.765757 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:25.765758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:25.765932 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:25.766231 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:25.766232 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:25.766232 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:25.766288 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:25.766759 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:32:25.766760 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:32:25.766820 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 03:32:25.766911 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 03:32:25.767022 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 03:32:25.767325 392 external_mini_cluster.cc:1428] Setting key b2500b1b3366fc5d679c7a1fb8ca29ef
I20260927 03:32:25.767367 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.757247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55852 (local address 127.0.98.60:43467)
0927 03:32:25.757886 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:25.757895 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:25.757927 (+ 32us) server_negotiation.cc:407] Connection header received
0927 03:32:25.758531 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:25.758540 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:25.758611 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:25.758761 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:25.759039 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.760431 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.761394 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.761958 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.762039 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:25.763879 (+ 1840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:25.763887 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:25.763887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:25.763906 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:25.765659 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:25.766033 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:25.766035 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:25.766036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:25.766152 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:25.766357 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:25.766358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:25.766359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:25.766663 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:25.766745 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:25.766986 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 03:32:25.767123 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":531,"threads_started":1}
I20260927 03:32:25.768808 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.760862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38772 (local address 127.0.98.62:42863)
0927 03:32:25.760928 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:25.760930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:25.761071 (+ 141us) server_negotiation.cc:407] Connection header received
0927 03:32:25.761098 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:25.761099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:25.761109 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:25.761181 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:25.761661 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.762229 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.763101 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.763308 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.763389 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:25.765769 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:25.765770 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:25.765770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:25.765778 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:25.767327 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:25.767994 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:25.767995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:25.767995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:25.768048 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:25.768277 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:25.768278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:25.768278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:25.768389 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:25.768456 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:25.768642 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 03:32:25.768718 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 03:32:25.768848 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.760778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42863 (local address 127.0.0.1:38772)
0927 03:32:25.760940 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:25.760976 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 03:32:25.761047 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:32:25.761193 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:32:25.761195 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:32:25.761497 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:32:25.761579 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:25.761580 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.762330 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.762331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:25.762988 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:25.762990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.763444 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.763445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:25.763521 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:25.764002 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:32:25.764007 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:32:25.765680 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:32:25.767664 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:25.767670 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:25.767672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:25.767931 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:25.768155 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:25.768156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:25.768157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:25.768219 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:25.768461 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:32:25.768468 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:32:25.768515 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 03:32:25.768612 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 03:32:25.768718 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":107,"threads_started":1}
I20260927 03:32:25.769276 23148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3462c6548024245a05897392fad6967: Bootstrap starting.
I20260927 03:32:25.769414 23223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46741 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:32:25.770279 23148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3462c6548024245a05897392fad6967: Neither blocks nor log segments found. Creating new log.
I20260927 03:32:25.770568 23148 log.cc:824] T 00000000000000000000000000000000 P c3462c6548024245a05897392fad6967: Log is configured to *not* fsync() on all Append() calls
I20260927 03:32:25.771648 23148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3462c6548024245a05897392fad6967: No bootstrap required, opened a new log
Sep 27 03:32:25 dist-test-slave-j0hf krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 03:32:25 dist-test-slave-j0hf krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 03:32:25.773571 23148 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3462c6548024245a05897392fad6967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28951365811142f8ac5b001938e52de0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "c3462c6548024245a05897392fad6967" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46741 } } peers { permanent_uuid: "7f9e785f54a940868a1d6a88f626a064" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43467 } }
I20260927 03:32:25.773727 23148 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3462c6548024245a05897392fad6967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3462c6548024245a05897392fad6967, State: Initialized, Role: FOLLOWER
I20260927 03:32:25.773820 23148 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3462c6548024245a05897392fad6967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28951365811142f8ac5b001938e52de0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "c3462c6548024245a05897392fad6967" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46741 } } peers { permanent_uuid: "7f9e785f54a940868a1d6a88f626a064" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43467 } }
I20260927 03:32:25.774187 23148 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3462c6548024245a05897392fad6967 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:32:25.774240 23229 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3462c6548024245a05897392fad6967 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28951365811142f8ac5b001938e52de0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "c3462c6548024245a05897392fad6967" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46741 } } peers { permanent_uuid: "7f9e785f54a940868a1d6a88f626a064" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43467 } } }
I20260927 03:32:25.774304 23229 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3462c6548024245a05897392fad6967 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 03:32:25.776410 23240 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c3462c6548024245a05897392fad6967: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:32:25.776460 23240 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:32:25.777068 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.769676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46741 (local address 127.0.0.1:52950)
0927 03:32:25.769758 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:25.769769 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:32:25.769812 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:32:25.769934 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:32:25.769935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:32:25.770066 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:32:25.770116 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:25.770118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.770816 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.770817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:25.771363 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:25.771366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.771876 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.771876 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:25.771938 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:25.772358 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:32:25.772363 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:32:25.773821 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:32:25.775825 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:25.775826 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:25.775826 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:25.776177 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:25.776390 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:25.776391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:25.776391 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:25.776449 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:25.776780 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:32:25.776781 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:32:25.776821 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 03:32:25.776879 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 03:32:25.776963 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 03:32:25.777437 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.769723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52950 (local address 127.0.98.61:46741)
0927 03:32:25.769762 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:25.769764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:25.769799 (+ 35us) server_negotiation.cc:407] Connection header received
0927 03:32:25.769867 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:25.769868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:25.769877 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:25.769931 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:25.770190 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.770741 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.771486 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.771756 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.771801 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:25.773920 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:25.773921 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:25.773922 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:25.773928 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:25.775718 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:25.776263 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:25.776264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:25.776265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:25.776322 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:25.776520 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:25.776521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:25.776522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:25.776679 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:25.776742 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:25.777278 (+ 536us) server_negotiation.cc:299] Negotiation successful
0927 03:32:25.777350 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:32:25.777734 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.767875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55858 (local address 127.0.98.60:43467)
0927 03:32:25.767910 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:25.767912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:25.768130 (+ 218us) server_negotiation.cc:407] Connection header received
0927 03:32:25.768150 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:25.768151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:25.768160 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:25.768211 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:25.768936 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.769609 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.770711 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.770995 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.771059 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:25.774011 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:25.774012 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:25.774012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:25.774018 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:25.776266 (+ 2248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:25.776641 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:25.776642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:25.776642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:25.776699 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:25.776902 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:25.776903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:25.776903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:25.777034 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:25.777184 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:25.777561 (+ 377us) server_negotiation.cc:299] Negotiation successful
0927 03:32:25.777628 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:32:25.777732 23223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43467 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:32:25.776758 23080 master_runner.cc:186] Error getting master registration for 127.0.98.62:42863: OK, instance_id { permanent_uuid: "28951365811142f8ac5b001938e52de0" instance_seqno: 1790479945456601 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 03:32:25.780534 23080 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 27 03:32:25 dist-test-slave-j0hf krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 03:32:25 dist-test-slave-j0hf krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 03:32:25.786096 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.777992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43467 (local address 127.0.0.1:55862)
0927 03:32:25.778162 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:25.778171 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:32:25.778207 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:32:25.778348 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:32:25.778349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:32:25.778481 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:32:25.778514 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:25.778516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.779345 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.779346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:25.780120 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:25.780123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.780552 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.780553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:25.780610 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:25.781060 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:32:25.781064 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:32:25.782447 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:32:25.784295 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:25.784296 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:25.784296 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:25.784484 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:25.784674 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:25.784675 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:25.784675 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:25.784744 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:25.785846 (+ 1102us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:32:25.785847 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:32:25.785881 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 03:32:25.785924 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 03:32:25.786004 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260927 03:32:25.786376 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.778135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55862 (local address 127.0.98.60:43467)
0927 03:32:25.778262 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:25.778263 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:25.778268 (+ 5us) server_negotiation.cc:407] Connection header received
0927 03:32:25.778282 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:25.778282 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:25.778289 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:25.778574 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:25.778592 (+ 18us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.779260 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.780207 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.780476 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.780747 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:25.782546 (+ 1799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:25.782547 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:25.782547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:25.782554 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:25.784207 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:25.784559 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:25.784560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:25.784560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:25.784616 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:25.785590 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:25.785591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:25.785591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:25.785765 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:25.786201 (+ 436us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:25.786239 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 03:32:25.786301 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260927 03:32:25.786720 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.778523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38788 (local address 127.0.98.62:42863)
0927 03:32:25.778557 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:25.778560 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:25.778628 (+ 68us) server_negotiation.cc:407] Connection header received
0927 03:32:25.778717 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:25.778718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:25.778727 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:25.778754 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:25.779380 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.780137 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.780966 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.781243 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.781313 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:25.783809 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:25.783810 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:25.783810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:25.783816 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:25.785490 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:25.785811 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:25.785812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:25.785813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:25.785866 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:25.786086 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:25.786087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:25.786087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:25.786224 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:25.786368 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:25.786585 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 03:32:25.786663 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:32:25.787752 23223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f9e785f54a940868a1d6a88f626a064: Bootstrap starting.
I20260927 03:32:25.788888 23223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f9e785f54a940868a1d6a88f626a064: Neither blocks nor log segments found. Creating new log.
I20260927 03:32:25.789175 23223 log.cc:824] T 00000000000000000000000000000000 P 7f9e785f54a940868a1d6a88f626a064: Log is configured to *not* fsync() on all Append() calls
I20260927 03:32:25.789994 23223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f9e785f54a940868a1d6a88f626a064: No bootstrap required, opened a new log
Sep 27 03:32:25 dist-test-slave-j0hf krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 03:32:25.792390 23223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f9e785f54a940868a1d6a88f626a064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28951365811142f8ac5b001938e52de0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "c3462c6548024245a05897392fad6967" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46741 } } peers { permanent_uuid: "7f9e785f54a940868a1d6a88f626a064" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43467 } }
I20260927 03:32:25.792546 23223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f9e785f54a940868a1d6a88f626a064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f9e785f54a940868a1d6a88f626a064, State: Initialized, Role: FOLLOWER
I20260927 03:32:25.792644 23223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f9e785f54a940868a1d6a88f626a064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28951365811142f8ac5b001938e52de0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "c3462c6548024245a05897392fad6967" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46741 } } peers { permanent_uuid: "7f9e785f54a940868a1d6a88f626a064" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43467 } }
I20260927 03:32:25.792991 23223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f9e785f54a940868a1d6a88f626a064 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 03:32:25.794783 23254 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7f9e785f54a940868a1d6a88f626a064: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:32:25.794874 23254 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:32:25.796624 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.787526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52958 (local address 127.0.98.61:46741)
0927 03:32:25.787572 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:25.787574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:25.787582 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:32:25.787602 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:25.787603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:25.787611 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:25.787666 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:25.788180 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.788705 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.789495 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.789804 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.789870 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:25.792535 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:25.792536 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:25.792536 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:25.792543 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:25.794282 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:25.795514 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:25.795515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:25.795515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:25.795584 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:25.795800 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:25.795801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:25.795801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:25.795920 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:25.796060 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:25.796446 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 03:32:25.796517 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 03:32:25.794967 23155 master_runner.cc:186] Error getting master registration for 127.0.98.62:42863: OK, instance_id { permanent_uuid: "28951365811142f8ac5b001938e52de0" instance_seqno: 1790479945456601 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 03:32:25.797688 23155 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 03:32:25.798053 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:32:25.798434 23073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43467 } attempt: 2
I20260927 03:32:25.799305 23047 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "28951365811142f8ac5b001938e52de0"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38788
Sep 27 03:32:25 dist-test-slave-j0hf krb5kdc[22992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790479945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 03:32:25.803592 23243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f9e785f54a940868a1d6a88f626a064 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28951365811142f8ac5b001938e52de0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "c3462c6548024245a05897392fad6967" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46741 } } peers { permanent_uuid: "7f9e785f54a940868a1d6a88f626a064" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43467 } } }
I20260927 03:32:25.803988 23243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f9e785f54a940868a1d6a88f626a064 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:32:25.806195 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.798647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43467 (local address 127.0.0.1:55878)
0927 03:32:25.798745 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 03:32:25.798756 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:32:25.798795 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:32:25.798969 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:32:25.798970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:32:25.799125 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:32:25.799162 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:25.799163 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.800039 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.800040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:25.800871 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:32:25.800873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.801452 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.801453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:32:25.801509 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:25.802011 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:32:25.802017 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:32:25.803384 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:32:25.805173 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:25.805174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:25.805174 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:25.805404 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:25.805610 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:32:25.805611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:32:25.805611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:32:25.805667 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:32:25.805922 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:32:25.805923 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:32:25.805962 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 03:32:25.806020 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 03:32:25.806105 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 03:32:25.806929 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.798763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55878 (local address 127.0.98.60:43467)
0927 03:32:25.798857 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:25.798859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:25.798866 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:32:25.798883 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:25.798884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:25.798892 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:25.798935 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:25.799225 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.799946 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.801040 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.801335 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.801396 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:25.803520 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:25.803521 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:25.803522 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:25.803528 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:25.805084 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:25.805477 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:25.805478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:25.805478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:25.805528 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:25.805729 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:25.805730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:25.805731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:25.805843 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:25.805897 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:25.806763 (+ 866us) server_negotiation.cc:299] Negotiation successful
0927 03:32:25.806849 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 03:32:25.808410 23073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28951365811142f8ac5b001938e52de0: Bootstrap starting.
I20260927 03:32:25.809093 23073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28951365811142f8ac5b001938e52de0: Neither blocks nor log segments found. Creating new log.
I20260927 03:32:25.809401 23073 log.cc:824] T 00000000000000000000000000000000 P 28951365811142f8ac5b001938e52de0: Log is configured to *not* fsync() on all Append() calls
I20260927 03:32:25.810205 23073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28951365811142f8ac5b001938e52de0: No bootstrap required, opened a new log
I20260927 03:32:25.811553 23073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28951365811142f8ac5b001938e52de0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28951365811142f8ac5b001938e52de0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "c3462c6548024245a05897392fad6967" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46741 } } peers { permanent_uuid: "7f9e785f54a940868a1d6a88f626a064" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43467 } }
I20260927 03:32:25.811694 23073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28951365811142f8ac5b001938e52de0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28951365811142f8ac5b001938e52de0, State: Initialized, Role: FOLLOWER
I20260927 03:32:25.811782 23073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28951365811142f8ac5b001938e52de0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28951365811142f8ac5b001938e52de0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "c3462c6548024245a05897392fad6967" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46741 } } peers { permanent_uuid: "7f9e785f54a940868a1d6a88f626a064" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43467 } }
I20260927 03:32:25.812073 23073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28951365811142f8ac5b001938e52de0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:32:25.813360 23047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28951365811142f8ac5b001938e52de0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 03:32:25.813427 23047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28951365811142f8ac5b001938e52de0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:32:25.814148 23047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28951365811142f8ac5b001938e52de0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28951365811142f8ac5b001938e52de0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "c3462c6548024245a05897392fad6967" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46741 } } peers { permanent_uuid: "7f9e785f54a940868a1d6a88f626a064" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43467 } }
I20260927 03:32:25.814392 23047 leader_election.cc:288] T 00000000000000000000000000000000 P 28951365811142f8ac5b001938e52de0 [CANDIDATE]: Term 1 election: Requested vote from peers c3462c6548024245a05897392fad6967 (127.0.98.61:46741), 7f9e785f54a940868a1d6a88f626a064 (127.0.98.60:43467)
I20260927 03:32:25.814743 23122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "28951365811142f8ac5b001938e52de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c3462c6548024245a05897392fad6967"
I20260927 03:32:25.815276 23122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3462c6548024245a05897392fad6967 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:32:25.815546 23197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "28951365811142f8ac5b001938e52de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7f9e785f54a940868a1d6a88f626a064"
I20260927 03:32:25.815665 23197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f9e785f54a940868a1d6a88f626a064 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:32:25.816215 23122 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c3462c6548024245a05897392fad6967 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28951365811142f8ac5b001938e52de0 in term 1.
I20260927 03:32:25.816479 23197 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7f9e785f54a940868a1d6a88f626a064 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28951365811142f8ac5b001938e52de0 in term 1.
I20260927 03:32:25.816702 23021 leader_election.cc:302] T 00000000000000000000000000000000 P 28951365811142f8ac5b001938e52de0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28951365811142f8ac5b001938e52de0, c3462c6548024245a05897392fad6967; no voters:
I20260927 03:32:25.812029 23257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28951365811142f8ac5b001938e52de0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28951365811142f8ac5b001938e52de0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "c3462c6548024245a05897392fad6967" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46741 } } peers { permanent_uuid: "7f9e785f54a940868a1d6a88f626a064" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43467 } } }
I20260927 03:32:25.817075 23257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28951365811142f8ac5b001938e52de0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:32:25.817142 23257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28951365811142f8ac5b001938e52de0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:32:25.817181 23257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28951365811142f8ac5b001938e52de0 [term 1 LEADER]: Becoming Leader. State: Replica: 28951365811142f8ac5b001938e52de0, State: Running, Role: LEADER
I20260927 03:32:25.817250 23257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28951365811142f8ac5b001938e52de0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28951365811142f8ac5b001938e52de0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "c3462c6548024245a05897392fad6967" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46741 } } peers { permanent_uuid: "7f9e785f54a940868a1d6a88f626a064" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43467 } }
I20260927 03:32:25.818238 23257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28951365811142f8ac5b001938e52de0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28951365811142f8ac5b001938e52de0. Latest consensus state: current_term: 1 leader_uuid: "28951365811142f8ac5b001938e52de0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28951365811142f8ac5b001938e52de0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "c3462c6548024245a05897392fad6967" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46741 } } peers { permanent_uuid: "7f9e785f54a940868a1d6a88f626a064" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43467 } } }
I20260927 03:32:25.818312 23257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28951365811142f8ac5b001938e52de0 [sys.catalog]: This master's current role is: LEADER
W20260927 03:32:25.819375 23268 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 28951365811142f8ac5b001938e52de0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:32:25.819454 23268 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:32:25.820550 23275 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:32:25.820715 23275 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:32:25.828646 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.819702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52964 (local address 127.0.98.61:46741)
0927 03:32:25.819744 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:25.819746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:25.820416 (+ 670us) server_negotiation.cc:407] Connection header received
0927 03:32:25.820511 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:25.820512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:25.820522 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:25.820561 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:25.821413 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.822162 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.823210 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.823541 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.823593 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:25.825227 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:25.825228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:25.825228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:25.825235 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:25.827177 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:25.827605 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:25.827606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:25.827607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:25.827660 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:25.827865 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:25.827866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:25.827866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:25.827982 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:25.828060 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:25.828471 (+ 411us) server_negotiation.cc:299] Negotiation successful
0927 03:32:25.828557 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:32:25.829555 23197 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7f9e785f54a940868a1d6a88f626a064 [term 1 FOLLOWER]: Refusing update from remote peer 28951365811142f8ac5b001938e52de0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:32:25.829802 23122 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c3462c6548024245a05897392fad6967 [term 1 FOLLOWER]: Refusing update from remote peer 28951365811142f8ac5b001938e52de0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:32:25.830053 23257 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 28951365811142f8ac5b001938e52de0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3462c6548024245a05897392fad6967" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:32:25.830232 23257 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 28951365811142f8ac5b001938e52de0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f9e785f54a940868a1d6a88f626a064" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:32:25.831486 23229 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3462c6548024245a05897392fad6967 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28951365811142f8ac5b001938e52de0. Latest consensus state: current_term: 1 leader_uuid: "28951365811142f8ac5b001938e52de0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28951365811142f8ac5b001938e52de0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "c3462c6548024245a05897392fad6967" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46741 } } peers { permanent_uuid: "7f9e785f54a940868a1d6a88f626a064" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43467 } } }
I20260927 03:32:25.831574 23229 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3462c6548024245a05897392fad6967 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:32:25.831629 23243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f9e785f54a940868a1d6a88f626a064 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28951365811142f8ac5b001938e52de0. Latest consensus state: current_term: 1 leader_uuid: "28951365811142f8ac5b001938e52de0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28951365811142f8ac5b001938e52de0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "c3462c6548024245a05897392fad6967" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46741 } } peers { permanent_uuid: "7f9e785f54a940868a1d6a88f626a064" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43467 } } }
I20260927 03:32:25.831705 23243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f9e785f54a940868a1d6a88f626a064 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:32:25.835928 23273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28951365811142f8ac5b001938e52de0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "28951365811142f8ac5b001938e52de0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28951365811142f8ac5b001938e52de0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "c3462c6548024245a05897392fad6967" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46741 } } peers { permanent_uuid: "7f9e785f54a940868a1d6a88f626a064" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43467 } } }
I20260927 03:32:25.836024 23273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28951365811142f8ac5b001938e52de0 [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:25.836241 23257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28951365811142f8ac5b001938e52de0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "28951365811142f8ac5b001938e52de0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28951365811142f8ac5b001938e52de0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "c3462c6548024245a05897392fad6967" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46741 } } peers { permanent_uuid: "7f9e785f54a940868a1d6a88f626a064" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43467 } } }
I20260927 03:32:25.836318 23257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28951365811142f8ac5b001938e52de0 [sys.catalog]: This master's current role is: LEADER
I20260927 03:32:25.836769 23243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f9e785f54a940868a1d6a88f626a064 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "28951365811142f8ac5b001938e52de0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28951365811142f8ac5b001938e52de0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "c3462c6548024245a05897392fad6967" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46741 } } peers { permanent_uuid: "7f9e785f54a940868a1d6a88f626a064" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43467 } } }
I20260927 03:32:25.836858 23243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f9e785f54a940868a1d6a88f626a064 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:32:25.836917 23229 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3462c6548024245a05897392fad6967 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "28951365811142f8ac5b001938e52de0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28951365811142f8ac5b001938e52de0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42863 } } peers { permanent_uuid: "c3462c6548024245a05897392fad6967" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46741 } } peers { permanent_uuid: "7f9e785f54a940868a1d6a88f626a064" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43467 } } }
I20260927 03:32:25.837007 23229 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3462c6548024245a05897392fad6967 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:32:25.837215 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.821563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38800 (local address 127.0.98.62:42863)
0927 03:32:25.821600 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:25.821602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:25.826382 (+ 4780us) server_negotiation.cc:407] Connection header received
0927 03:32:25.826477 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:25.826478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:25.826487 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:25.826516 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:25.828704 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.829347 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.831787 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.832082 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.832150 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:25.833489 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:25.833490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:25.833490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:25.833496 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:25.835131 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:25.835823 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:25.835824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:25.835824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:25.835887 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:25.836505 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:25.836506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:25.836506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:25.836661 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:25.836749 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:25.837051 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 03:32:25.837130 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:32:25.836947 23281 mvcc.cc:205] Tried to move back new op lower bound from 7333805858114826240 to 7333805858068340736. Current Snapshot: MvccSnapshot[applied={T|T < 7333805858114826240}]
I20260927 03:32:25.838140 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:25.820184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55892 (local address 127.0.98.60:43467)
0927 03:32:25.820240 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:25.820242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:25.823556 (+ 3314us) server_negotiation.cc:407] Connection header received
0927 03:32:25.823658 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:25.823659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:25.823668 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:25.823700 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:25.824467 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.825157 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.826099 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:25.826430 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:25.826494 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:25.835274 (+ 8780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:32:25.835276 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:32:25.835276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:32:25.835283 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:32:25.836941 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:25.837350 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:25.837351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:25.837351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:25.837404 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:32:25.837590 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:32:25.837590 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:32:25.837591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:32:25.837722 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:32:25.837788 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:25.837982 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 03:32:25.838052 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 03:32:25.839912 23275 catalog_manager.cc:1347] Generated new cluster ID: f8dd64410c51462b84e58218cf78951d
I20260927 03:32:25.839957 23275 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:32:25.843502 23282 mvcc.cc:205] Tried to move back new op lower bound from 7333805858114826240 to 7333805858068340736. Current Snapshot: MvccSnapshot[applied={T|T < 7333805858114826240}]
I20260927 03:32:25.854729 23275 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:32:25.855566 23275 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:32:25.864769 23275 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28951365811142f8ac5b001938e52de0: Generated new TSK 0
I20260927 03:32:25.865006 23275 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 03:32:25.889369 23076 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:32:25.888299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38814 (local address 127.0.98.62:42863)
0927 03:32:25.888344 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:25.888346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:25.888707 (+ 361us) server_negotiation.cc:407] Connection header received
0927 03:32:25.888811 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:25.888812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:25.888823 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:25.888853 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:25.889303 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38814: BlockingRecv error: recv got EOF from 127.0.0.1:38814 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 03:32:25.890786 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:32:25.889941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52980 (local address 127.0.98.61:46741)
0927 03:32:25.889978 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:25.889980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:25.890274 (+ 294us) server_negotiation.cc:407] Connection header received
0927 03:32:25.890363 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:25.890364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:25.890375 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:25.890404 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:25.890724 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52980: BlockingRecv error: recv got EOF from 127.0.0.1:52980 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 03:32:25.891736 23225 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:32:25.890159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55904 (local address 127.0.98.60:43467)
0927 03:32:25.890195 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:25.890197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:25.891225 (+ 1028us) server_negotiation.cc:407] Connection header received
0927 03:32:25.891318 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:25.891319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:25.891329 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:25.891358 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:25.891682 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55904: BlockingRecv error: recv got EOF from 127.0.0.1:55904 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 03:32:25.894405 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:32:25.893464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52994 (local address 127.0.98.61:46741)
0927 03:32:25.893501 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:25.893503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:25.893770 (+ 267us) server_negotiation.cc:407] Connection header received
0927 03:32:25.893878 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:25.893879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:25.893915 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:25.893948 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:25.894365 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52994: BlockingRecv error: recv got EOF from 127.0.0.1:52994 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 03:32:25.897462 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:32:25.896731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53006 (local address 127.0.98.61:46741)
0927 03:32:25.896762 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:25.896764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:25.896979 (+ 215us) server_negotiation.cc:407] Connection header received
0927 03:32:25.897074 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:25.897075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:25.897087 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:25.897115 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:25.897418 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53006: BlockingRecv error: recv got EOF from 127.0.0.1:53006 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 03:32:25.910619 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:32:25.908022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53022 (local address 127.0.98.61:46741)
0927 03:32:25.908061 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:25.908063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:25.910060 (+ 1997us) server_negotiation.cc:407] Connection header received
0927 03:32:25.910164 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:25.910165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:25.910176 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:25.910206 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:25.910565 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53022: BlockingRecv error: recv got EOF from 127.0.0.1:53022 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 03:32:25.920611 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:32:25.919758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53030 (local address 127.0.98.61:46741)
0927 03:32:25.919796 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:25.919797 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:25.920049 (+ 252us) server_negotiation.cc:407] Connection header received
0927 03:32:25.920151 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:25.920152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:25.920163 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:25.920192 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:25.920536 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53030: BlockingRecv error: recv got EOF from 127.0.0.1:53030 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 03:32:25.930845 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:32:25.929979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53040 (local address 127.0.98.61:46741)
0927 03:32:25.930019 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:25.930021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:25.930269 (+ 248us) server_negotiation.cc:407] Connection header received
0927 03:32:25.930370 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:25.930371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:25.930382 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:25.930412 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:25.930788 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53040: BlockingRecv error: recv got EOF from 127.0.0.1:53040 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 03:32:25.949275 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:32:25.948362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53042 (local address 127.0.98.61:46741)
0927 03:32:25.948406 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:25.948408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:25.948672 (+ 264us) server_negotiation.cc:407] Connection header received
0927 03:32:25.948781 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:25.948782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:25.948795 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:25.948826 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:25.949216 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53042: BlockingRecv error: recv got EOF from 127.0.0.1:53042 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 03:32:25.984565 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:32:25.983547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53046 (local address 127.0.98.61:46741)
0927 03:32:25.983600 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:25.983602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:25.983744 (+ 142us) server_negotiation.cc:407] Connection header received
0927 03:32:25.983829 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:25.983830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:25.983954 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:25.984091 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:25.984484 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53046: BlockingRecv error: recv got EOF from 127.0.0.1:53046 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 03:32:26.050573 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:32:26.049771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53054 (local address 127.0.98.61:46741)
0927 03:32:26.049864 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:26.049866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:26.049959 (+ 93us) server_negotiation.cc:407] Connection header received
0927 03:32:26.050007 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:26.050008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:26.050019 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:26.050110 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:26.050492 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53054: BlockingRecv error: recv got EOF from 127.0.0.1:53054 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 03:32:26.180480 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:32:26.179790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53070 (local address 127.0.98.61:46741)
0927 03:32:26.179842 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:26.179844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:26.179949 (+ 105us) server_negotiation.cc:407] Connection header received
0927 03:32:26.179979 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:26.179980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:26.179993 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:26.180048 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:26.180395 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53070: BlockingRecv error: recv got EOF from 127.0.0.1:53070 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 03:32:26.438290 23153 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:32:26.437637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44814 (local address 127.0.98.61:46741)
0927 03:32:26.437692 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:26.437694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:26.437704 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:32:26.437777 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:26.437778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:26.437789 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:26.437844 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:32:26.438200 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44814: BlockingRecv error: recv got EOF from 127.0.0.1:44814 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:32:26.776901 23240 catalog_manager.cc:1259] Loaded cluster ID: f8dd64410c51462b84e58218cf78951d
I20260927 03:32:26.776971 23240 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c3462c6548024245a05897392fad6967: loading cluster ID for follower catalog manager: success
I20260927 03:32:26.777806 23240 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c3462c6548024245a05897392fad6967: acquiring CA information for follower catalog manager: success
I20260927 03:32:26.778132 23240 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c3462c6548024245a05897392fad6967: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 03:32:26.795325 23254 catalog_manager.cc:1259] Loaded cluster ID: f8dd64410c51462b84e58218cf78951d
I20260927 03:32:26.795426 23254 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7f9e785f54a940868a1d6a88f626a064: loading cluster ID for follower catalog manager: success
I20260927 03:32:26.796267 23254 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7f9e785f54a940868a1d6a88f626a064: acquiring CA information for follower catalog manager: success
I20260927 03:32:26.796586 23254 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7f9e785f54a940868a1d6a88f626a064: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 03:32:26.954272 23351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:26.951466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44826 (local address 127.0.98.61:46741)
0927 03:32:26.951587 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:26.951589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:26.951660 (+ 71us) server_negotiation.cc:407] Connection header received
0927 03:32:26.951775 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:26.951776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:26.951841 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:26.951931 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:32:26.952088 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:26.952661 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:26.953426 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:26.953640 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:26.953713 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:26.953890 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:32:26.954012 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:32:26.954096 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:26.954122 (+ 26us) server_negotiation.cc:299] Negotiation successful
0927 03:32:26.954143 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":55,"threads_started":1}
I20260927 03:32:26.958252 23357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:26.955998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41764 (local address 127.0.98.60:43467)
0927 03:32:26.956113 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:26.956115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:26.956166 (+ 51us) server_negotiation.cc:407] Connection header received
0927 03:32:26.956227 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:26.956227 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:26.956261 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:26.956329 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:32:26.956419 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:26.956897 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:26.957560 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:26.957756 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:26.957808 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:26.957932 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:32:26.958026 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:32:26.958081 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:26.958115 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 03:32:26.958132 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":57,"threads_started":1}
I20260927 03:32:26.962338 23351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:26.959983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44830 (local address 127.0.98.61:46741)
0927 03:32:26.960038 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:26.960039 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:26.960047 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:32:26.960094 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:26.960095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:26.960106 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:26.960179 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:32:26.960325 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:26.960782 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:26.961648 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:26.961849 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:26.961922 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:26.962096 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:32:26.962135 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:32:26.962188 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:26.962248 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 03:32:26.962268 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:32:26.962831 23357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:26.960127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41770 (local address 127.0.98.60:43467)
0927 03:32:26.960183 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:26.960184 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:26.960288 (+ 104us) server_negotiation.cc:407] Connection header received
0927 03:32:26.960371 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:26.960371 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:26.960381 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:26.960424 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:32:26.960522 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:26.960971 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:26.962036 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:26.962334 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:26.962394 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:26.962586 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:32:26.962613 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:32:26.962638 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:26.962741 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 03:32:26.962763 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 03:32:26.963866 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:32:26.959953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33242 (local address 127.0.98.62:42863)
0927 03:32:26.960254 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0927 03:32:26.960256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:32:26.960265 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:32:26.960920 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:32:26.960921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:32:26.960968 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:32:26.961046 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:32:26.961232 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:26.961983 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:26.962942 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:32:26.963270 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:32:26.963327 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:32:26.963527 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:32:26.963627 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:32:26.963706 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 03:32:26.963759 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 03:32:26.963783 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":49,"threads_started":1}
I20260927 03:32:26.971055 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 23005
I20260927 03:32:26.976848 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 23080
I20260927 03:32:26.981740 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7YYkzb/build/release/bin/kudu with pid 23155
2026-09-27T03:32:26Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2812 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5608 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (271123 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST